Of Software Abstraction and Salt

"Too much salt in food and it will taste bitter, too little and it may taste bland"

The above analogy was given during my code review 10 years back by experienced colleague in context of a real time data collector module made using C Programming Language and pthread. I went on to refactor the module. Simplified the project layout, removed data structures, global state machine based code branching using an array of function pointers, configuration that was making the software difficult to understand in entirety.

Imagine if you have a team of Django/Python developers of 25 with only 3 Developers who are experienced and rest freshers. You choose to build a web application in Python that uses

  • Django/Python
  • Django cookie cutter
  • Dockers Containers
  • Django Rest Framework based API
  • redis for caching
  • Ansible scripts for automating deployments
  • Over 3 dozen Django Third Party App ( this doesn't include dependencies )
  • Custom middleware doing everything from logging, analytics, caching
  • Making Django Signals way too many things it shouldn't be doing
  • Two NoSQL for storing different type of information.
  • Not using Django template but jinja
  • Not using Django models
  • Used a obscure queuing system because some article was trending on hacker news

Software abstraction is like salt, sprinkle the right amount and what you have is a tasty meal.

What is the right amount ?.

  • Have good decoupling of your components and clear definition of responsibility of each layer in your architecture.

  • Understanding of Where to use Data abstraction ? and Where to use Control flow abstraction ?

  • Introducing more abstraction to solve workflow, execution flow issues due to concurrency / not encapsulating user actions / issues that require correct selection of algorithm/data structure is like rubbing salt on wounds.

  • Be considerate of the skillset, team composition, training and onboarding required to get everyone on the same page. May be have a consistent hiring policy.

Architecting Software

Watched ETL Is Dead, Long Live Streams - By Neha Narkhede a few days back. Highly recommended for anyone who created an ETL system. I worked on batch oriented ETL with end results being a solr instance with 20-30 Million documents using hadoop for transformation. Thanks to that work I could understand and appreciate this talk by Neha Narkhede. I haven't used Kafka but intend to. Have looked at spark recently. After watching the talk one thought stuck. A Good Architect is one who designs the system such that it stands the test of time. If one looks at his/her software architecture after say 2 years and feels 40% to 60% of software design / tool selection decisions have not scaled as per expectation or become bottleneck. Then the architect is certainly doing a disservice. One way to get things right is to read extensively / research, ask people smarter then you. Not just one person but multiple. Also it's important that one be willing to not seek refuge in the language, tool, framework that he/she knows. Have some ETL working coming up, looking to put spark, kafka to work.

UFC 205 Notes and Prediction

  • Eddie Alvarez looks pretty much on edge with a little provocation. I think it may be his undoing in the ring,
  • Stephen Thompson is a amazing guy in terms of composure. #Respect to his responses and the way he carries himself. Also he is very skilled who overcame health issues like dominic cruz. Def a good role model.
  • When you think cornor can't make it more interesting and exciting with his talk he takes it to the next level. His best ability is to switch off when he gets in the ring. At least that is how it looks.


  • Eddie Alvarez vs. Conor McGregor

    Winner - Eddie (I am rooting for Conor)

  • Tyron Woodley vs. Stephen Thompson

    Winner - Stephen Thompson (I am rooting for Stephen)

  • Joanna Jedrzejczyk vs. Karolina Kowalkiewicz

    Winner - Joanna (I am rooting for Karolina)

  • Chris Weidman vs. Yoel Romero

    Winner - Chris (I am rooting for Chris)

  • Miesha Tate vs. Raquel Pennington

    Winner - Pennington (I am rooting for Pennington)

  • Khabib Nurmagomedov vs. Michael Johnson

    Winner - It's KhabibTime

Things one notices as weekly newsletter curator

The Good

  • People writing brilliant stuff that blows your mind away, inspires you to better your skills

  • You start detecting/noticing trends in technology / programming language ecosystem before others do

  • You get to know people who are the doers / community enablers

  • You notice people who started writing on basic topics and after a year of honing their craft graduate to developers with good skillset. The growth is a sight to behold

The Ugly

  • People submitting articles that has plagiarism written all over it. They are summary of chapter of a book you read / created from an 2-3 blog post on the same subject

  • Content Marketers hiding dates on articles / recycling old blog post by changing dates

Notes from pycon india

  • If you think Python 3 has 20%-30% share in new Python coded software you are sleeping. It's probably 50% to 60%.
  • I used AWS EMR/hadoop with mrjobs for some work recently. After attending the talk on Spark by Shagun it became clear how superior Spark is.
  • There is a huge interest in programming community on
    • Data Science
    • AI and Machine Learning
    • NLP
  • Python is a mainstream programming language. Be it 10th to 12th standard students to engineering college students or dozens of ex Java programmers who are now coding in it. Everyone is on the Python bandwagon. And this is a good thing.
  • Every year only 2-3 talks at Pycon are relevant rest don't fall in my area of interest or are beginners oriented or too verbose on textbook gyan with very little code to back it up.
  • I gave a hurried up flash talk on "Five Modules Every Python Developer Should Know". Having participated a lot in Debates, Elocution it's pretty clear I need to get back into the game by speaking at user groups and meetups.
  • I think workshops are better then talks from gaining actual skills. Talks are great for high level overviews or demos.
  • This might be my last Pycon India conference. I did my bit with spearheading Mumbai Python User Group. My personal learning goals and ambitions don't align well with Pycon India. Personally I learned more interacting with online communities then offline/real world these days.

Upcoming ImportPython Changes

Was at #pyconindia and got to meet over 2 dozen subscribers of http://importpython.com

Here is what they liked about the newsletter

  • Curation with description/excerpt explaining the article,

  • It's a toss up on github projects I curate. 50% want to focus on popular / trending projects as oppose to what I do that's focus on ones that were released last week. The other half love the fact that they find projects that are simple scripts / tools they borrow from,

  • Knowing which event is happening is awesome. But need to make it compact. Also focus needs to be on user group meetups,

  • Those who subscribed to python subreddit mentioned the other newsletters to be less interesting because mostly all their articles were already read by them through rss reader. We feature a lot of stuff that hasn't made it to reddit yet makes it interesting.

What are the new things being planned based on the feedback received ?

  • New Improved Newsletter Design ( built from ground up using foundation's email framework mostly ),

  • Better Typography,

  • Render well in mobile email clients,

  • I intend to make sections fluid i.e. ability to introduce new sections as and when needed as oppose to fixed ones,

  • Ability to search entire newsletter archive as one feedback was how tedious it was to dig articles from previous issues,

  • Exploring other forms of content presentations i.e. gif, video, embedding tweets.

  • Exploring more social platforms.

Here we go again

Come 1st Sept will be giving another shot at being an entreprenuer. I intend to focus on email newsletters and everything around them. My experience of running http://importpython.com for the last 2 years is the reason behind the decision.

It's not like I have an idea that would change the landscape of newsletters. Just that I see myself making more or less the same money as my current salary so it makes sense to be my own boss.

One thing you would see me do more is blog.


Shantanu turned 3 years old on 21st May. He has an outgoing, social personality not a shy, awkward, cranky kid. It's a right age for me to introduce him to physical activties, music. I don't want to make the classic mistake of putting a kid though a lot of stuff/activites as they grow and the net gain is nothing.

My mother while I was growing up observed my passion for computers and supported me by getting all the books I wanted to read, a computer and whatever it's that I wanted to advance my passion and knowledge.

Shantanu's young obviously for this, but point being to observe carefully what a child likes and get him to do more of it, enjoy and get better at it.

Nothing like supporting your child in what he likes and loves.

It's not what's in the Machine that matters.

2010 I wanted to buy a laptop, my age old desktop had crashed. Had only 30,000 INR / 500$ in my current account and believe me I was desperate to get a computer again. Went to croma ( electronics store in Mumbai ) and ask the attendant.

Me: Which is the cheapest laptop you have ?
Attendant : Sir we have this lot of lenovo rejected by a corporate company since it doesn't have any OS and they wanted Windows. We are selling it for 28,000 INR while it's original price is 36,000 INR. 

The configuration was decent

  • Dual Core Intel Processor
  • 4 GB RAM
  • 15 inch screen
  • Integrated sound, mic, camera.

I did the best work of my life till date on that machine.
Wrote three software products end to end on it, won 3 hackathons. Used the laptop till 2014-2015 and gave it to my sister. She used it for a year until it's keyboard stopped working.

Later in 2012 brought a mac mini and plugged it with a gorgeous 19 inch Dell monitor. I occasionally use it.

Being a programmer I tend to get excited with everything tech, that includes new shiny mac hardware aka macbooks.
Came pretty close to buying one like 10 times now. Justification in my head ?

  • Latest and greatest of the hardware and software inside it.
  • How I will use it to usher a change in my life.
  • Most of my programmer friends have one.

Everytime I think of buying one I stop because of the lesson the cheap lenovo ThinkPad laptop taught me.

It's not what's in the Machine that matters. I need to find that motivation and drive again, a macbook ain't gonna fix it.

When Quantity Beats Quality - Content Production Boxing vs MMA.

Big fan of Boxing and newbie fan of MMA.

Boxing Shows are produced by HBO and Showtime i.e. 24x7 and All Access respectively. They produce one/two episodes that promote the match. The shows are of high quality (Emmy award winning quality). There is no denying that some of the world's most talented people produce these shows. Here is an example of 24x7 show

MMA is mostly UFC. UFC does a number of small 5-15 minute shows free on YouTube and some aired on Fox channel before the match. Approx 30-40 videos a month are uploaded regularly to promote different fighters / different cards. The quality of the video, script, cinematography is definetly run of the mill. They produce a lot more content with meagre resourses unlike network cable companies like HBO and Showtime.

Today UFC is thriving while Boxing is stagnent. Both serve the same market, fans who love combat sports.
UFC executive understand well the profile of the people who watch combat sport better then the cable executive companies. Read the script of voice over guy in HBO videos. Whomsoever wrote the script has an outstanding command over English, so much that an average combat sports fan would refer a dictionary to understand some words. The sentence formations and prose in HBO's 24x7 would make a literary agent jump out of his couch and stalk the guy who wrote the script.
UFC on the other hand keeps it simple and give more content per fight, in a simple language and thus offer more insight into the surroundings and background of the fighter.

This is how goliath is beating david. Quantity resulting is more engagement vis a vis Quality.

Watch Your Watch

Mamaji ( mother's brother ) took me to Titan showroom to get a watch. I choose the same watch that my cousin was wearing those days, this is in 1996-1997. It was a Timex watch, water proof - 100m. The watch had a minimalist dial with the minute, seconds arm, the face just had only a single silver dot for 12 o'clock against a black background. I slept with it, bath with it, gave all my exams with it. Can never recall taking the watch off my wrist till 2005, and when I did I lost it.

After that I have tried an expensive watch, two casio classic digital watch and haven't been able to have the same relationship as with the first one.

This post is about my pursuit of finding a new watch.

What's the purpose of a watch ?

To see the time and date. That's about it.

Now that's where I am wrong. If you ask people why they are willing to pay an exorbitant amount of money for watches the answers are

a) I like how it looks / I like the design,
b) I spotted a colleagues / friends / relatives wearing an X brand and I want to do it too.

Why not today watches are also fashion accessory. Nothing wrong with it.

The problem is people have no idea on What's inside a watch ?. Or Why is this watch 10x more expensive that the other watch ?. Their watch is of which type for that matter.

This video talks about the three types of watch that one would encounter most

A brief description on Chronograph

please do ask yourself if stop watch is of utility to you in your day to day job/life ?.

Why Swiss watches are expensive and What's Swiss made vs Swiss movement watch ?

I am wearing this beauty

Got myself a manual mechanical watch ... Beautiful and simple dial.

A photo posted by Ankur Gupta (@originalankur) on

It has a simple classic dial a manual hand wind mechanical watch from HMT.

Last week I tried the 38mm apple smartwatch and it looks like I might have finally found the one that would fill the hole timex left.


A Programmer spend a whole lot of time using the text editor. I have never been a power user of any text editor. When it comes to terminal it's vim. For frontend developement it's Webstorm. For Python development it's PyCharm. For markdown, todos, notes etc it's sublime. For diff I used diffmerge by sourcegear. Infact at a given point of time I have atleast 2-3 editors opened. For the last week I have been comparing sublime to atom and looking to becoming a power user in one editor.

a) Find code/function/variable usage,
b) Navigate code from one module to another,
c) Refactor,
d) Commit from inside the editor,
e) Json view,
f) Visualize commit history of file(s),
g) Compare same file side to side,
h) PEP8 compliance and linter,
i) Build markdown files from inside the editor,
j) Do most tasks without having to use the mouse.
k) Setup a basic frontend development setup with syntax highlighting, navigating divs using keyboard,
l) Get a static analysis tool to integrate and show errors before committing.

Hopefully by next week I will write a basic intro to Atom and how to accomplish the above tasks.

Books To Read - 2016 Q1

MySQL Developers Library

Been writing lot of SQL queries for the last year as part of my Job at Intellus Learning

Streaming Data

Haven't really worked at scale when it comes to streaming data etc. So looking to learn a thing or two from this book.

Fluent Python

This is an amazing book for Python Programmers. It offers a deeper explaination on pretty much everything in Python.

Becoming a Better Programmer

Long time back I read this online mini-book of sort by a retired computer programmer of mainframe era on "How to become a better programmer". I don't have the link but it left an impression on me. Based on the sample chapter I believe this to be a good book to read.

jQuery in Action

So last I coded in AngularJS for a UI. Have to work on a personal project and planning to use jQuery for it. I know most would consider it as a step backward, but I think it's the right tool for the job.

The Test Prep App fad in India

In Edutech space test prep app are the new fad.

Market Size

Theoretically this is a virgin territory waiting to be disrupted. Coaching classes that train students to ace competitive examination cost a lot of money, not tailor made keeping a individual student's competency. On the other hand test prep apps cost 1/10 th of the cost and adapt based on which topic user is incompetent in. For a country like India ( 1.2+ Billion population ) this looks like a huge market. Education is important to parents and the willingness to pay is there.

Technology Stack

Tech behind test prep app is simple CRUD application, with few machine learning algorithms thrown in for either adaptive learning / user personalisation. This clubbed with a question bank puts you in business. No wonder you have coaching classes launching their own apps.


The ground reality is adoption rate ( paid subscriptions ) are low and the only chest thumping number are the no of questions attended/attempted, total users ( made up of above 90% - 95% free tier users ), page views. In US the emerging trend is that content publishers who have been publishing test prep books are the only ones gaining and holding on to their market share. The reason is brand recall, strong team of academia that have been working with the same companies for years, generation of unique question bank each year. Unlike e-commerce where it's speculated that winner takes it all, I believe there is not going to be a loss leader in test prep app for another 5-7 years. Why ? Because parents who eventually pay for these apps believe a traditional model of classroom, and don't want to experiment with their child's future. They do however acknowledge the complementary utility of these apps. However students going to classes have little time to spare on day to day basis thanks to hours spend in school and coaching classes. This fact clearly shows up in the paid user's monthly usage reports.

The above facts emerged from conversation with friends who works for Two VC funded test prep app companies in India. I reached out to them because one of these companies founder reached out to lead the Python/Django team.

The funding craze in India ( angel/seed level ) will see more test prep startups. Each being funded primarly on the user engagement and potential market size. User engagement metrics fueled by free users and market size by looking at coaching class market size. The metrics are misleading and the market is refusing to be disrupted as of now.


So Shabda an friend and founder of agiliq.com has donated djangoweekly.com to importpython.

April first week will see launch of a new newsletter focussed on Django.

There we go again

After having blogged for 7+ years I decided to deleted my old weblog and start again.
So here we go.