Fool me twice

On Tuesday, November the 3rd 2020, as I sat eagerly to watch the results of the Presidential Election, very quickly my hope of seeing the country get rid of Trump turned into despair. In the first few minutes, the Florida results were encouraging. Then Miami-Dade county reported and the shock I felt will remain forever etched in my memory. Hillary Clinton won that district by 35 points, and still lost Florida. Biden won it by around 15 points or so, and it was immediately clear that he is not going to win Florida.

As they say, it was “deja vu all over again”. Four years ago, it was Florida that delivered the first shock and things got progressively worse throughout the night - for people like me who wanted to see the country rejecting Trump. This year, it seemed like a replay of the same movie. Both times, the pain became worse because I wasn’t prepared for it. The polls had left me no doubt that Trump was going to be defeated, and both times, the polls got it horribly wrong.

In 2016, when Trump won, the polls became a joke in the eyes of the most. I was also surprised how the polls got it wrong. I decided to spend time figuring out why. As I studied the issue, I realized that I didn’t really understand what the polls were actually saying. Of course, the polls got many things wrong, but their error was pretty much in line with their historical performance.

Even in 2012, the polls had underestimated Obama’s chances of winning reelection. But they got the direction right, and hence the polling error did not get too much attention. The polls were actually not as wrong in 2016, but the error margin was bigger than the slim vote margin that went in Trump’s favor. More importantly, Trump’s margin of victory was far less than the “undecided” percentage of voters reported by polls. That was the key to understand why we all though the polls were wrong. As the election neared, there was a clear movement of these “undecided” in Trump’s favor, which many missed, including the media and me. Then there was the last minute bombshell by Comey which sealed the deal, and there were few polls after that announcement. 

Many theories such as “the shy Trump voters” have been debunked in many studies since the 2016 debacle. The real reason, the pollsters argued, was that they didn’t account for the correct proportion of rural voters, people without a college degree etc. In other words, their sampling technique needed refinement to capture a better representation of the electorate.

So I defended the polls and prediction models such as the one at FiveThirtyEight. I argued that a few swing states, going in Trump’s favor by a very slim margin, is definitely within their prediction of a 30% probability of Trump’s win. In life, we shouldn't be surprised by events with 30% probability actually happening. For example, we are not at all surprised to see a pair of dice rolling to 6 on each, which has a much less probability (less than 3%). 

This time, I thought I was interpreting the polls correctly. The polls were indeed much better in 2018 midterms. In 2020 as time progressed, they tended to move more strongly in favor of Biden. That was not the case 4 years ago when the polls were fluctuating a lot. There were hardly any “undecided” in the polls this time, and Biden’s margin was much much higher than Clinton. To top it, the polls were consistent with each other, across different pollsters, across states etc. 

So I made my prediction for a Biden win, and doubled down on it by giving a 50-50 chance for a landslide victory by Biden. I even offered the bets with specific odds. Although I did not lose the bets, and got a few things right, I was wrong about lot more things.

Now the way ballots were counted, made it seem like a close and tough win. Actually it was quite a comfortable victory for Biden. When Trump won it last time, many called it a “clear mandate”. Biden’s 2020 electoral count is going to be the same as Trump in 2016. Biden’s popular vote margin (of close to 7 million votes) is even larger than what Clinton got. By the way, my estimation of the margin was quite right. Biden’s margin in swing states is much much higher than what Trump had in 2016. Finally, Biden won back the Blue states Clinton lost, and flipped 2 Red states in addition. So by any yardstick, this is a very nice victory. Just the way counting was done, made it look like a tough win.

In spite of that, I have to say that my prediction was quite wrong.

Most of the arguments I made were based on eroding support for Trump. That was completely incorrect. Trump got even more votes than he got the last time (close to 10 million more), and was able to get more votes from Blacks and Latinos than he got in 2016. There is a lot for me to understand there.

Mainly, I wasn’t predicting just a Biden win. I was giving a 50-50 chance for a landslide. As it turned out, there was really no chance of a landslide. I made that prediction thinking Florida will go to Biden and Texas will be a close loss. This was based on polls. For example in Florida, the polls continued to shift towards Biden as the election neared. Most highly rated polls gave Biden a +5 chance, when the reality was he was -3. That was an astronomically huge miss by the polls. When most swing states are decided by a margin of less than 2%, any poll that can be wrong by 8-10% is worse than being useless.

Florida wasn’t an exception. Iowa, Texas, North Carolina - none were as close as the polls claimed to be. And wherever Biden won, the polls had estimated a much higher margin.

The common theme in the errors made by the polls was - they underestimated the support for Trump. Just like they did in 2016. Except this time, the pollsters had told us that they have learned from their mistakes and have made the necessary adjustments. If this is how anyone “improves” in their job, they would be fired.

Fool me once, shame on you. Fool me twice, shame on me.

Hence, I am declaring that polling is useless for me. Yes, they will try to explain why they got it wrong this time, or how it wasn’t as wrong as we perceive. Yes, they might get it right the next time. But I am not going to trust them again. Ever. And I know, I am not alone. Even though Biden won as was the prediction, many people understand that polls were really inaccurate this time - especially in the purple swing states where they matter the most. The entire polling industry has lost its trust. Even if they get it right the next time, it won’t increase their trustworthiness any more than throwing a dice.

Nah. I did not make a good prediction. Live and learn. Well, at least I won’t waste any time and suffer heartburn in the next election, by engaging in studying polls and coming up with a prediction model. 

My Prediction For 2020 Election

The case for an epic landslide Democratic win

The 2020 Presidential election is only a few days away. Many questions are being discussed. Will there be a peaceful transfer of power if Biden wins? Will the polls be wrong again for predicting a Trump loss? And so on.

What is rarely mentioned is the possibility of an epic loss for Trump. Some believe that Teflon Trump will emerge victorious as he did in the 2016 primaries and eventual election. Some believe that his die hard supporters (or cult, as it’s often called) are the most loyal of all. Some believe polls are no longer reliable. And so on.

Almost an year ago, in late 2019, in my friend circles, I went out on a limb and outlined my prediction for a “yuuuge loss” for Donald Trump. At that time, it was even more of a contrarian opinion. Incumbents have an advantage. A good economy is a virtual guarantee that the incumbent will win again. Still I had my reasons to make that prediction.

But then the pandemic happened. Since I found it completely out of the “analyzable” space, I retracted my prediction, saying, I have no idea how this will play out. Especially with the rapid spread, and both the candidates being over 70 and in the high risk group, in April I had no idea what to expect going forward.

Now, I am again comfortable to reissue my prediction with an updated probability. Trump is going to suffer an epic loss.

Before I give my reasons, let’s define what my prediction means.

What’s in epic loss? I consider getting less than 200 electoral votes as epic loss. Or a blowout. So my prediction is Trump does not win more than 200 electoral votes.

How confident am I in making this prediction? Since the future is unknowable, we can only talk in terms of probabilities. I give it a 50-50 chance. Now before you smirk, let me restate. This is my estimate for the probability that Trump loses the election AND fails to win over 200 electoral votes. The probability for a general Biden win is much higher in my estimate. In other words, I am willing to have a bet with a friend, where if my prediction comes true, I win a dollar and if Trump gets more than 200 electoral votes I give them a dollar even if Trump still loses the election. For a general election result, I am willing to give a 1 to 3 payout ratio. Meaning, if Trump wins, I give them 3 dollars, and if Trump loses, I get only 1 dollar.

What are my reasons for such a high level of confidence?

Here is how I divide the eligible voters into distinct subsets.

1. People who always come out to vote, and vote Democratic.

2. People who always come out to vote, and vote Republican.

3. People who always come out to vote, and vote based on their views on that specific election - the Independents.

4. People who need motivation to come out to vote and if they do, they vote Democratic. 

5. People who need motivation to come out to vote and if they do, they vote Republican.

6. People who need motivation to come out, but may vote either way depending on the election.

7. Net new additions to the voter pool based on old people dying, and young people becoming eligible to vote and will vote. Some may vote for Republicans, some may vote for Democrats.

8. People who never come out to vote. Unfortunately there are such irresponsible people in every democratic country.

Which groups are more important in deciding the election? Without knowing the sizes, it’s hard to say. For example, if group 1 or 2 is 90% of the country, it really doesn’t matter what other groups feel or do. Thankfully, that’s not the case.

Since we don’t know the absolute sizes of each subset, we can then focus on estimating the change - the delta - between the last election and this. We can ask how do we expect these groups to behave differently in this election than what they did the last time. It’s a valid strategy to ask that question, as at least one candidate is the same in both the elections. So we can frame this question with respect to Trump. 

Let’s examine one at a time and focus only on the delta.

Group 1. By definition this group will come out and will vote Democrat, so no delta.

Group 2. By definition this group will come out and will vote Republican, so no delta.

Group 3. This group is more interesting to speculate about. Remember, this group will come out and vote. If they vote the same as last time, there is no delta. I am very certain that’s not the case. This is a referendum on the incumbent. Last time, some from this group voted for Clinton, and some voted for Trump. If someone was willing to vote for Clinton last time, I am sure they will vote against Trump again. Those who voted for Trump in this group, are more likely to change their minds. Not all. But some or many. Remember again, this group doesn’t represent the cult of Trump in any meaningful way. All of that is accounted for in group 2. We are only interested in the delta. So, I am arguing, this group represents a big negative delta for Trump. In other words, far more people in this group will change from Trump to “no on Trump”. Please note, not all have to change their mind this way. Some might actually change their vote from Democratic in 2016, to Trump in 2020. I am speculating about the aggregate, not the individual. So “net negative” is the operative word.

Group 4. This was a strong reason for Clinton’s loss. Many people who would have voted for a Democratic candidate, simply did not show up. This is a combination of die hard Sanders supporters and people who disliked Clinton. I am arguing they have a very strong motivation to come out and vote for the Democrats. They see Trump’s record. They see what’s happening with the Supreme Court. Biden is not as disliked as Clinton. Again, not all will come out to vote. We are just looking at the delta. This is a positive delta for the Democrats. No doubts about it.

Group 5. The reverse of group 4. This will be another big negative delta for Trump. How many are going to be really enthusiastic to come and re-elect Trump? A reminder once again. The cult of Trump is already accounted for in group 2. Coming back to this group of “likely Republicans”, will all of them come out to vote again for Trump? When there is no Clinton to defeat? Just like group 4 sitting at home in 2016, a section of this group will sit this one out. This is another negative delta for Trump.

Group 6. For someone who needs a strong motivation to come out and vote for someone (either Democrat or Republican), what’s their motivation this time? If they were not motivated last time to vote for Trump, they are not going to be motivated this time! In fact some may be motivated to vote against Trump. That’s why I am giving this a net negative delta for Trump.

Group 7. Another interesting group to speculate about. But maybe easier to get an agreement on. New young voters - which way will they go? Is there any doubt at all? Seriously! This is another net negative delta for Trump. And perhaps the biggest delta in percentage terms, when compared to the group size.

Group 8. By definition, there will be no delta here.

So there you have it. I do not know the size of any delta. I don't need to. Because none of the subsets is generating a positive delta for Trump. If there was a positive delta, then the question of the size becomes important. Because one positive delta can in theory override all other negative deltas. But I am confident that each delta is negative. Some mildly negative, to some wildly negative. Hence my prediction, and the confidence.

You are going to see this delta affecting “down ballot” candidates. This is not only going to go against Trump in a big way, but against Republicans in general. While I don’t think Lindsay Graham will actually lose, or that Texas will actually turn blue this time, both are a lot closer to happening than last time. As a side effect, Democrats have a good chance of getting complete control - the Presidency (high probability), the House (near certainty) and the Senate (slightly better than a 50-50 chance).

<TL;DR>

All this can be simplified into one question. Has Trump won more supporters than what he has lost? If you think he has converted more to support him than what he has lost, feel free to have a bet with me. Remember again, don’t just look at his cult. It’s the delta that’s going to decide the election, like it does in all the elections. His cult has not grown. But the size of his detractors has grown considerably.

Last time, he won with a razor thin margin. That margin got amplified, as it does in many elections, due to the structure of the electoral college. But that works in the other way too! If he loses that razor thin margin, the electoral map will look a lot different this time. I am betting, since he will lose a lot more than thin margin, it’s going to look like a blowout - less than 200 electoral votes.

UPDATE October 15, 2020

A couple of friends pointed out that I have not called out some of the data points clearly. So here is more explanation - no change in the forecast.

First, this way of doing analysis based on speculating about delta, is valid only for incumbents, which is the case here. Second all the deltas must be all negative, or positive else, the size of the deltas matter, and those are really hard to estimate.

Third, it’s valid only in cases where the margin was not high when the last election was won by the incumbent. I should have called out the third point very clearly. For example, President Obama did not get as many votes in his reelection bid, as he got the first time. Still he won the second time. In spite of negative delta. But his margin of victory was large enough to cushion it. That’s not the case with Trump.

Let’s look at the data. In Michigan, Trump won by an extremely small margin of 11K votes. That’s 0.23% of the votes. That margin is extraordinarily hard to defend, especially when you see that, in 2016 all other candidates (Green Part, Libertarian etc) got 5% of the vote (250K). So the chances of negative delta making Trump lose Michigan is very high. Note that Michigan had voted Democratic in the previous 6 elections. So 2016 with such a thin margin is not a secular change, but an outlier.

In Wisconsin, Trump won by 23K votes, which is about 0.75%. Third party candidates won 5%. And this state has voted Democratic for the previous 7 elections. This was again an outlier, which is very hard to defend for Trump.

In Pennsylvania, Trump won by 44K votes, which is also about 0.75%. Third party candidates got over 3%. This state has also voted Democratic in the previous 6 elections.

This is the basis for speculating about the deltas. A margin of less than 1% in a Democratic stronghold is too thin to stand on for a Republican candidate.

Similarly, Trump's margin of victory in other states was - Florida 1.2% and both Arizona and North Carolina less than 4%. None of these states is reliably Republican.

Conservatives, Liberals and Logic

These days, “How can they be so illogical?”, is one of the most common questions I hear people asking in exasperation. Sometimes, “illogical” is replaced with “dumb”, “stupid” etc, but the intent is really to question the “abilities of deduction”. The other variants of this question are “How can they be so evil?”, “How can they be so wrong and not know it?” and so on. In this post, I am only going to talk about the “illogical” nature of “them”. 

I really find that question funny, because most of the time, people asking that question really don’t know what “Logic” means. Let’s revisit something that all of us have learned in middle school.

Logic helps us decide, given a set of premises, if the conclusion of inferences is correct. In other words, it helps us appraise an argument. Using logic, we can objectively assess an argument as either valid or invalid.

We all learned Geometry in school. Euclid constructed the most beautiful edifice in human knowledge, by starting with extremely simple postulates and using them to prove theorems of  increasing complexity. Everything about Euclid’s geometry is absolutely logical.

Except his postulates. They have no logical proof. They are taken as “self evident truths”. For example, one postulates roughly states that it is possible to draw a straight line from any one point to another. There is no way to prove this statement. We assume it to be true and use it for logically proving other theorems. As long as these “axioms” feel trivially true, everyone will be comfortable using them as the starting point.

But what if the axiom doesn’t feel trivially true? That indeed was the problem with Euclid’s 5th, and last postulate about parallel lines. This is a big topic, and I can refer you to Euclid’s Window. By changing the fifth postulate, mathematicians were able to come with completely new Geometries, that are very different but as logical as Euclidean Geometry. As it turns out, they are far more than mathematical curiosities, and the space in which our stars and galaxies operate, indeed follows non-Euclidean Geometry.

What does all this have to do with conservatives and liberals and their logical or illogical arguments? The point is, if the starting premise is different, then the logical argument would lead to a very different conclusion. Duh, you say. OK. So here is the crux. There is simply NO way to logically choose a premise. That whole thing - the inferences, the deductions - that whole logic thing, is what comes afterwards. 

So, it’s totally possible that “they” are making a very logical argument, but “their” premises are vastly different than yours. Here is what you probably don’t want to hear - “their” premises are not worse or better than your premises - they are just different premises.

This is not just a semantic jugglery. This is also not an attempt to whitewash by saying, “everyone is correct”. No. There indeed are a lot of stupid people who make illogical arguments. Their premises are contradictory in nature, and/or their inferences do not follow the rules of Logic. But questions such as “What ought to be” come down to morality, and cannot be settled by logic. Our moral code is driven by our intuitions, which are the result of our evolution by natural selection. Again, a big contentious topic and I can refer you to The Righteous Mind.

This is also not to say that all debates with “them” are on morality. Not at all. For example, when it comes to public policy, effectiveness (usefulness v/s harm) can often be debated objectively, using data analysis. That’s another big topic, as statistical interpretations can also get clouded by confirmation bias and frankly, dishonest intentions.

So now what? How can you convince “them” that “their” argument is illogical? Seriously, if you are still asking that question, you need to read this post again. Instead of looking at the argument, why not look at the premise of that argument? Why not examine it to see if it can be derived from even more fundamental premises, or is it truly subjective?

More importantly, why the need to prove to “them” that “they” are morally wrong? If Mathematics can have internally consistent theories that contradict each other, what’s wrong in a society of humans who hold very contradictory but logical views? Of course it’s not comfortable, but that diversity of thoughts and opinions, you know the thing that we say we celebrate - how about actually celebrating that?

The Undoing Project

Book Review : The Undoing Project

Author : Michael Lewis

My Rating : 4 out of 5 stars

The complete title of the book is “The Undoing Project: A Friendship that Changed Our Minds”.

Michael Lewis is one of the most famous authors of our time. He has written many best sellers, and a few of his books have been made into successful movies. His latest book tackles a subject that is perhaps far more fundamental than his other books.

The scientific advancements that have happened in last few decades have had a huge impact on our understanding of the world. That includes not just about the universe and subatomic particles, but also about the evolution and our own mind. How our mind thinks, and how it comes to conclusion is a topic that interests me greatly. I am not talking about the anatomy of brain, but the thought process and its implications.

A big contribution to the very foundation of what we know today, was laid in 70s and 80s by two scientists in Israel - Daniel Kahneman and Amos Tversky. Their work on coming up with a model of decision making  - “Prospect Theory” as they called it - is one of the cornerstones of a new field that emerged from it - “Behavioral Economics”. Kahneman won a Nobel Prize in Economics in 2002 (Tversky had passed away then, and Nobel prizes are not awarded posthumously). Later other economists have also won the Nobel Prize due to their contribution to Behavioral Economics. This is field that intersects both Economics and Psychology.

This book tells the story of how these two very different personalities came to collaborate together in Israel. It starts at their childhood, their early careers in Israel of the 60s, how the wars impacted them and how they came to be collaborators. The author also goes into many details to explain their personalities and their friendship. A lot of research must have been done to gather this information, and kudos to the author.

The book tries to be both a biographical account and a primer on their work. It works very well, although the focus in mainly on the human story. The scientific work and the explanation is interspersed with the life stories. I think it’s very well done. This way the reader is not overwhelmed with the technical details, and the biography doesn’t remain just a story of two extraordinarily talented people. The aim is to also convey how their work came into existence and how seminal it it was, and the book succeeds.

I could have given this 5 stars, and some readers may very well do so. There are two reasons for my not giving 5 stars to this book. The biography part, on a few occasions, felt repetitive. Especially, about how like Amos Tversky was. After a few times, I was saying, yes, yes, yes, we get it - He was a popular figure. Second reason is, the explanation of the ideas in Prospect Theory is good, not great. I won’t recommend this as a popular science book. If you want to read how we handle uncertainty then I can recommend to you The Drunkard’s Walk, or The Improbability Principle. If you want to read about how we think, then I can recommend to you, The Righteous Mind and Some We Love, Some We Hate and Some We Eat. None of these are on Behavioral Economics, but they do a great job of explaining their respective ideas.

With that said, I can definitely recommend this book. It’s very readable, has great story-telling and has enough educational material. It’s a story of two great minds, whose work is profoundly impacting Economics and something worth knowing about.

Rising Out Of Hatred

Book Review : Rising out of Hatred

Author : Eli Saslow

My Rating : 5 out of 5

The complete title of the book is “Rising out of Hatred : The Awakening of a Former White Nationalist”.

Just 3-4 years ago, a well known thought leader of White Nationalism, Derek Black, renounced his cause and turned into a liberal. He wasn’t just some leader, he was looked at as “The” future leader. As a son of a former Grand Wizard of the Klu Klux Klan, as an eloquent speaker, very polished in debates, with a radio show that was widely popular among the followers, as the main speaker at their conferences, he was the rising star and universally loved by White Nationalists. And all this even before he was in college. In changing his positions, he had to publicly admit that he was wrong, and proclaim that the views he once espoused were dangerous. He had no choice but to go against his father, his family and cause them great emotional stress. He had to accept the danger that he will be labeled as a “traitor to the cause”. But he did the right thing. Because he clearly understood how illogical his ideas were, and how much harm they can cause. And he had to fight against those ideas.

This is an incredible story of transformation, courage and honesty. If I had watched such a story in a Hollywood movie, I would have most likely shaken my head in disbelief. “Such things don’t happen in real life”, is what I would have said. But reality is indeed stranger than fiction. That may be a cliche, but it’s true here.

I learned about this book on “The Daily Show” where both the Pulitzer Prize winning author Eli Saslow and Derek Black appeared together. I got so intrigued by the interview that I had to read this book, and I am glad I did.

The story is incredible, and Eli Saslow’s writing is smooth and gripping. This is not a story that can be written with politically safe language. It’s just not possible. This book tells things as they are. This is a history of last few years. As Derek Black was transforming himself from a White Nationalist into a liberal, at the same time, the Republican Party was transforming itself from a conservative party to Trump’s party. Many of the ideas of the White Nationalists which were considered fringe a few years ago, were becoming mainstream and foundational to the new Trump party. What the White Nationalists were hoping to happen for many many years, was suddenly happening with Trump (and of course his team) taking the charge. They were speaking the same language, expressing the same victimhood and proposing the same talking points. While Derek was giving up these ideas that he once promoted on his radio show, those exact same ideas were actually being taken up by the Presidential candidate of the Republican Party. 

These two separate but opposite transformations have been very superbly captured by the author. I am sure, given the current state of affairs, many people on the right, and especially Trump supporters would not like reading these details. But the author has no qualms in stating the facts and writing with complete candor. As many of us know and argue, he documents how the transformation of the Republican party started with the election of President Barack Obama. It was then aided by Fox News and completed by our current President Trump and his team.

It’s not just ironic, it horrified Derek. His transformation took place over the years when he was in college, with help from his unlikely friends which included one deeply religious Jewish friend and one immigrant from South America. His girlfriend was instrumental in challenging his views with facts after facts very patiently presented over many many months. But it was personal. He did not want to make a public scene out of it. But Donald Trump’s rise was the reason he decided to write editorials in liberal newspapers including The New York Times. He felt a personal responsibility to oppose the direction taken by the new politics of our country.

It is impossible to not feel respect for Derek Black. Very few people like to admit, even to themselves that they were wrong. To do it publicly, and going against their family is extremely rare. To do so at this extent - from a White Nationalist to a Liberal - is unheard of. And remember, all this in his twenties. It’s not just that he did it, but why he did it. As the book demonstrates, Derek changed his views, because of his analytical and highly intelligent mind. When presented with facts, it became clear to him that he was horribly wrong.

I cannot recommend this book enough. Especially if you are leaning on the Republican side of the spectrum and even more so, if you are a Trump supporter. No, I am not naive to think that anyone would change their mind by reading a book. But at least to some people, it will be help understand why many of us think that the politics of Trump is so dangerous to the fabric of our society. Well that’s my hope. Not expecting any agreement, just a bit of understanding.

Everybody Lies

Book Review : Everybody Lies

Author : Seth Stephens-Davidowitz

My Rating : 5 out of 5 stars

The complete title of the book is “Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are”.

Like I have done for some other book reviews, I must start this review with proper disclosures. I spent a significant portion of my career at Yahoo, where “Big Data” or “Data Science” or whatever you want to call it, is a big part of the strategy. I have myself worked on projects related to this field, and the results of my own code have astonished me by the sheer amount of insight that can be extracted out of sometimes simple techniques. I am not new to this field, and I am very familiar with the potency of such data mining exercises. 

So a lot of insights presented by this book were not surprising to me at all. That includes the techniques explained in the book for finding correlation using statistics, as well as the kinky nature of a lot that people search for. Like many software engineers do, almost a decade ago, I downloaded Apache Lucene to familiarize myself with the search engine technology. This download includes the sample of actual search terms used by internet users, and it felt that almost all the terms were pornographic in nature, and sometimes quite disturbing. So yes, I kind of knew a lot of the findings of this book already.

If you are not from a similar work background, be prepared to be surprised, enlightened, entertained, disturbed and a bit depressed. Sometimes, all at the same time.

In this fascinating book, the author will give you a grand tour what insights about specific individuals, as well as society in general can be inferred these days. The main source of this information is internet of course. As the book explains, the interaction people have with the various websites, and especially Google and PornHub are much much richer in terms of data mining, compared to the traditional mechanisms such as surveys. Of course the main reason is anonymity - as in people thinking that they are not revealing their identity. So they search for things and visit sites that they would not admit in polite company. They leave anonymous comments that can be analyzed using natural language processing techniques. The quantity of such interactions rise and fall with events that happen in real world. Such trends give far deeper insight into what people are actually feeling, which non-anonymous methods of interactions would simply not reveal.

All this is possible, because people behave differently in public and in private - especially when they are alone and think they are not being monitored. Hence the title of the book - Everybody Lies.

We all know that Facebook posts by users are nothing but advertisements (often false) about what they want their friends to perceive about their life. Reality is very different. But how different? By how much? And what about politically incorrect issues? For example, how racist people really are? What social policies do they really support? And so on.

The author gives a very nice introduction to techniques that have been used by companies, and individual researchers that have access to such internet interactions, what insights we can derive from this research and how valuable this can be.

I will be doing a terrible disservice to this book, if I give an impressions that everything is all dark and gloomy. Or this books is all about interactions on Google, Facebook and PornHub. The author himself takes pains to clarify that this is about “New Data” not just the size of the data. Also included are many many examples of non-internet data that can be as insightful.

This book is not a sales pitch for “Big Data”, and that’s great. You will also learn the limitations of these new techniques, and how mis-interpretations can happen. It’s all written in a very accessible way. No knowledge of economics or computer science is required. In fact, on few occasions, I felt that the author is trying to simplify it too much. Just a quick note : it’s clear that the author’s own stance on many political issues is towards the left. I am fine with it, but some readers may not appreciate it too much.

It’s a wonderful book, very relevant to what’s happening today. It will remind you of the great book “Freakonomics”. If you liked that one, you will like this one too. The findings are sometimes as you expect using common sense, but more often than not, they will be new insights. I highly recommend it.

Thoughts And Prayers

Sadly, these are the times of “Thoughts and Prayers”. The following are my hopes, thoughts and prayers.

When it comes to gun violence in USA, I am very pessimistic of any real change - in my lifetime. I think my pessimism is realistic and not emotional. The reasons for my pessimism have little to do with the NRA or the impotent and corrupt Congress. It’s the people. The completely brainwashed population. Just like a person suffering from a mental disorder who is unable to realize the disorder, the majority of the voting population doesn’t even know how twisted, irrational their thought process is. They will never realize. It’s an incurable disease. 

The debate has been distorted. It’s too late to even argue about who distorted this debate, and why. There is no way to settle this via rational debate. Nothing will be changed by pointing out the absurdities in pro-gun arguments. We have been having these debates forever. The Twitter memes have been going on forever. Nothing changed after Sandy Hook. Nothing meaningful will change now. Hence I think my pessimism is realistic.

But my pessimism is only for this generation. I am pessimistic to see a change in my lifetime. I do think eventually USA will become almost free of guns. I may not see it, but my prayer is, my great grandchildren will live in a safe country, where guns are simply not available. Where only military and police have access to guns. Just like most of the countries on this planet. Eventually, USA will emotionally be a mature country on the topic of guns. Eventually.

This requires a social evolution, which is a slow process. Social evolutions do not happen because people change their opinions. Most people never change their opinions. Social evolution happens simply because Time replaces existing people with new people. The cycle of death and birth, the foundation of biological evolution, is also the foundation of social evolution. Not the sole mechanism, but the foundation.

Every generation is more tolerant, more liberal than the previous one. On an average basis. The definition of Left and Right is subjective and changes in each generation. There will always be Left and Right, Liberal and Conservative, but the center point keeps shifting to left with every generation. I am talking about social policy, not fiscal policy. When it comes to racism, or the acceptance of gays, overall compassion, open mindedness and so on, the newer generation is simply better than its previous generation. Our kids are better human beings than us.

The same will be true for guns. The newer generation will not accept the current interpretation of the Second Amendment. They will make it relevant to their day and age. They will confiscate military assault weapons. They will make their society safer. We might add a minor new law, or tweak existing laws, but they will make drastic changes. That’s my prayer.

I pray that the newer generation realizes that the time for debate is over. It’s time to vote. Vote with their ballots, vote with their money. They are marching, giving passionate speeches, engaging the opposite side. That’s today. Tomorrow they will vote, and society will change, slowly, but surely, one vote at a time.

White Working Class

Book Review : White Working Class

Author : Joan C. Williams

My Rating : 3 out of 5

The complete title of the book is “White Working Class : Overcoming Class Cluelessness in America”.

After the financial crisis of 2008, there was a slew of books explaining what happened, why it happened and how a repeat can be prevented. It was exhausting to even look at the number of books on the same topic. A similar thing is happening after the 2016 election. Everyone is trying to explain the Trump victory which has been a crisis for most progressives. I was shocked myself, and wrote a blog post expressing my dismay.

Honestly I am getting tired of these explanations. They are getting very repetitive. It’s always the same theme - “how the liberal elitist globalists smugly ignored the middle class Americans devastated by changing economics who voted with their middle finger”. Not that it is wrong, it isn’t. But that’s not the complete picture, more about that in a minute.

So why did I pick up this book? The appeal was in the organization. One short chapter each for every important question. Author Williams does not shy away from asking direct questions. After explaining why we should talk about class, she first defines who is WWC (white working class). Then there are chapters on “Is the working class just racist”, “Why doesn’t the working class go to college” and so on. 

All the chapters are short. The author comes to the point quickly, directly and bluntly. It’s a very refreshing approach, irrespective of how much you agree with. These are tricky, controversial and often emotional issues. A complete agreement is nearly impossible. I agreed with most of the analysis, and I think most readers will. 

In spite of the agreements, I still have major objections to the approach. I know the author is trying to simplify, but dividing entire population in two groups, elites v/s working class, is completely lacking in nuances of the real world. Yes, a lot of people do fall in those two sets, but I don’t fall in either category, and I suspect many won’t. 

Another problem is putting all the blame on liberal elites. Of course there is blame, but this working class has made mistakes too. Many areas are deep red Republican, and have been for a very long time. Voting for the same candidate or the same party over and over again and getting angry about economic problems, indicates emotional voting patterns. The issue of police brutality is real. Not properly sympathizing with the angst represented by “Black Lives Matter” is unfair and results in more divide. We can disagree about the solution, but the refusal to admit Global Warming is laughable. So don’t blame the divide just on liberals.

At the same time, some real mistakes by liberals are simply ignored. The extreme liberal stance on immigration, which includes “sanctuary cities” is simply unpalatable to most Americans, myself included, and even to some liberals. The liberal hypocrisy, which routinely criticizes Christianity but fails to criticize far darker sides of Islam, is visible to everyone except the far left. Finally, the topic of guns gets no mention at all. Such omissions make an otherwise excellent explanation, incomplete.  

Relating this to 2016 election should have been done with other observations as well. Apart from the class divide there was another problem - namely Hillary Clinton - a candidate with a lot of baggage. It’s all speculation, but it’s possible to imagine Bernie Sanders winning against Trump, or even Governor Kasich winning against Hillary. I would have voted for Kasich. So not mentioning Hillary’s failure to stay out of suspicion is a flaw. Then praising Bill Clinton as someone who connected with the WWC, but not mentioning the role his legacy played in the election is a serious omission. Of course the class divide is real, but that wasn’t the only factor.

That’s what I meant above. Most such explanations focus only on the class divide, and fail to appreciate the chronic irrationality on both sides and severity of the flaws of the Democratic candidate. 

In the final chapters the author offers her thoughts on how the gap can be bridged. It’s all well intentioned, sometimes accurate but honestly, very weak in places. She is correct in pointing out that “higher minimum wage” is not what will solve economic plight of WWC. But saying that changing the slogan from “Pro-choice” to “Pro-child, pro-choice and pro-family” will have any impact, is just naive. 

It’s still a readable book. Don’t think of it as a complete analysis. Whatever has been analyzed, is insightful. That should be enough, as it’s small, feels just like a very long editorial and can be finished in one seating. 

My thoughts on : Tax Cuts And Jobs Act Of 2017

This is not a review of the new tax law (See full PDF) signed by President Trump today. This is just what I feel about the things that matter to me, and things that I feel are important.

Full disclosure on political views. I am neither a Republican, nor a Democrat and one of the “Never Trumper” guys. I had to vote for Hillary Clinton because of Trump’s candidacy. My views can generally be categorized as “social liberal, fiscal conservative”, but that’s not very accurate either. For example, I strongly oppose “Sanctuary Cities”, and I do want near-free college education for all. So there you go. Whatever prevalent label you attach me, will likely be wrong.

Full disclosure on fiscal views. 

Taxes are necessary. Duh, yes. The point is, “some” taxes are necessary. I do not know what the ideal tax rate should be, and that's not the point of this post.

Simple is good. Duh again, but important to list. Not just because of tax filing headaches, but also for cost of enforcement.

Deductions are unfair. Yes, including, mortgage interest deductions, and even 401K deferred taxes. Due to tax brackets, higher income people get more out of these deductions that lower income people. Most deductions and credits are not universal, they favor one group over another.

Double taxation is unfair. Yes, even when it happens to rich people. 

Now about the things that directly affect  me.

Mortgage interest deduction. I live in the San Francisco Bay Area, where housing prices are, and have always been in the stratosphere. Now mortgage interest only up to 750K can be deducted. It most likely will slow down the increases in home prices. That’s good. My house value may not go up as much, but it’s good for future home buyers. Affordable homes are good for the society. Primary residence is not an investment, or a lottery ticket to retire rich. I support this. 

Changes to AMT. I wish they had repealed AMT. But at least the new law takes a large section of the population out of AMT. My problem with AMT is not the actual taxes paid, but it makes taxes complicated. If higher taxes is what’s needed, increase the tax rate. Don’t complicate the tax code. Again, even if my effective taxes hadn’t changed, I would still support repealing AMT. Well, that didn’t happen, but this is still a big positive.

Changes to SALT deductions. The new law significantly limits how much state and local taxes can be deducted. This one hurts me. It’s very possible that due to AMT changes, tax rate reductions I might come out even. Regardless of the impact on me, I support this. Federal taxes are paid for services received from federal government. Why should state taxes be deductible any ways? Now this will get Californians angry. But I do think California taxes are too high, the state government is too big and there is a huge waste of taxpayers money. Maybe now, people will get more conscious about the taxes paid to their state government and demand lower rates, and more efficiency. It might make California less socialistic, and that would be a good thing. This kick in the gut was needed, and my only complaint is it does not go far enough.

Rolling back of tax rates. The tax rate reductions for individuals is not permanent. If deficit was an issue, then why cut taxes on high income earners? Why not make the tax rate cut permanent for lower income people? Maybe it’s political game of kicking the can down the road, and future congress may make the rate cuts permanent. With the current wealth gap, and a booming economy for the wealthy, the rate cuts should have been done only for the lower income people. I view this as a big negative in this tax law. 

Now for changes that do not directly affect me.

Corporate tax rate cut. Lowering corporate taxes is a good thing. I wish it was tied to job creation, or increased pay for lower salary grades. Especially considering the trillion dollar deficit that’s being added by this law. As it stands, companies most likely will return cash to shareholders via dividends, buybacks etc. That’s perfectly fine from shareholders’ point of view. The issue is these shareholders are mostly in the wealthier section of population. So the country as a whole gets a trillion dollar increase to the already high deficit, and the majority of the population gets nothing much to show for it. That’s my current feeling. If it does spur job growth, pay growth, well, that’s what the Republican congressmen and Trump’s team are promising. Only time will tell. So I am neutral on this right now.

Estate Tax. I will never be impacted by this, but just repeal that thing. If you need higher taxes, just increase tax rates. Do not double tax. The new law does not repeal it, but at least increases the threshold. It's not enough, but still a positive.

Questionable deals. There is a nice provision for pass through businesses, which may be needed. But the way it was added, raised lot of conerns. There is lot of information available on the CorkerKickbacks. Similarly there is a nice concession for private jet owners. That’s what politics has become today. People who write laws, make sure they get benefits out of it.

There is lot more in the tax code, but these are the points on which I have an opinion.

Summary. There is lot to like about this law. Whether the expected economic expansion happens and if it justifies the deficit or not, only time will tell. I am optimistic at this time.

What would be the political implications? This is definitely a great legislative victory for President Trump, and he has delivered on a big campaign promise. It’s also an accomplishment for both Rep Paul Ryan and Senator Mitch McConnell. It would very likely help the Republicans in 2018 midterm elections, because there will be a short term boost to the economy. The deficit concerns are much longer term concerns. The Republicans have taken a calculated risk, or a gamble if you will. I hope they are right and the benefits of this are reaped by lower income people as well. If not, there are always elections where we can vote for candidates that will undo these policies. Just like what the current administration is doing.

The ugly, the bad, and the (maybe) good

Four years ago I wrote about why President Obama's reelection made me happy. I hoped the Republican Party learnt a lesson from that defeat, and focuses more on issues than on hatred. I was hopeful. But exactly opposite happened. Hatred was very much alive in this election.

A lot has been written on why Trump has won, and many excellent articles are worth reading. Michael Moore wrote a prophetic article.  A fantastic blog post that went viral explained a lot. Now Matt Taibbi has written something very similar. 

I had read those first two links before election and I was still shell shocked at the result. I still am very sad about it. It’s not the sadness of the loss. Hillary Clinton was never really “my” candidate. It was Donald Trump who made me passionately wish for Hillary’s win. The sadness is from someone like Donald Trump actually winning it.

Elections are not just political events. They are social events too. In a way, they are the best survey of what the society really is. It’s very easy for me to accept that the majority did not agree with my political views. It’s a small tiny little price to pay for the privilege of living in a democracy. What is hard for me to accept is that half the society chose to elect a candidate that I think is morally unfit to be our President. His statements against minorities, immigrants, Muslims, women - everything has disgusted me. His crazy conspiracy theories, his Tweets, his campaigning style against the Republican candidates made me dislike him to the core. There is simply nothing about him that I find as a redeeming quality. Never had I despised a candidate so much.

I find this as the ugly side of this election result. As a society we chose to overlook some serious flaws in a person while handing over tremendous power to him. Not voting for Hillary is understandable, but voting for Trump is immoral. This is not a political statement. Disliking Trump is not same as liking Hillary. You can denounce both of them, but supporting Trump is self defeating. There has to be a minimum standard of decency a person has to clear before becoming the President. There has to be something that’s a deal-breaker. Not finding a single deal-breaker in Trump, when there were plenty, is the ugly ugly side of this election. This has given an insight into the society as no other single event has. I am saddened by the message we have given to ourselves, our kids and the entire world that - We simply have lost our moral compass. 

The danger is this ugly side surfacing itself in public behavior. Trump is not dumb. He might go back on anything and everything he has said, now that he has won. But his win is likely to embolden the dark side of human nature in our society.

Now the real bad side. The social damage this can do, potentially, is scary. The legislative and executive branch is in the control of one single party who will use that now to push a social agenda that’s not progressive. The supreme court will be nominated with judges who will very likely undo the work done so far. A section of population will rejoice, another section will weep.

Trump has promised to “drain the swamp”. If his ego-maniacal nature results into overlooking competent people in the Republican party and surrounding himself with novice sycophants and crackpots like Ben Carson, we will have a huge problem on hand. Will racial profiling come back? What happens to health and environmental regulations?

The other type of bad can happen even with right intentions. Repealing Obama Care is a done deal. But there are parts that are almost universally loved, for example, not denying coverage due to pre-existing conditions. It will be a tricky balance but may be achievable. Other promises are not at all easy to keep. The jobs that have been lost due to technological automation and globalization are not coming back. Putting tariffs will be counter-productive. Even with incentives for businesses, it’s hard to see how they can be competitive in a global marketplace. If these promises are broken, the whole negative voting cycle will repeat. 

It’s not all bleak. And it’s more than just the silver lining on the cloud.

First, I do not agree with the interpretation of the election, as some silent majority has spoken. The Trump supporters were not silent. Second, the majority did NOT vote for Trump. The majority voted for Hillary. Trump’s victory is anti-establishment, but the congress was still won by the establishment. Nothing much changed there.

Next, I do not think this was a victory of racism or xenophobia or misogyny or hatred. Yes the person elected has all those bad qualities, but the electorate doesn’t. Yes, some may have voted for those reasons. Just like in previous elections, some may have voted for Obama because he is black, and some against him because he is black. Both are wrong reasons. But it’s a small section for sure. The red states have always been red, the blue states have always been blue. The surprise came from MI, WI and PA. The rural white counties in these states voted for Trump this time, but had voted for Obama in previous elections. That’s not racism. They voted out of their economic frustration. I think they made a mistake, and Trump will make their bad situation worse. But they want to give him a chance, and it’s their choice to make, and it’s their democratic right. They voted for him, in spite of his bigotry, not because of it.

This vote for Trump, in my opinion, is more of a vote against Hillary. The democrats and liberals have to accept this fact, that their candidate was deeply flawed. Hillary knew for at least ten years that she was going to run for President. She should have made every effort to be above suspicion. She made bad decisions about her emails and her foundation. She paid the price. I do think she was a better candidate than Trump, but there is no reason to be blind to her flaws.

The real hope is the way next generation voted. I often feel that when it comes to social policy, the future generation is way better than my generation. They are far more inclusive. That’s the hope for progressive minded people. There is going to be another election in four years. Start the work. Find a better candidate. I know I am going to be far more politically aware of what’s going on from now. This election result has energized me, not made me feel hopeless. Hopefully those who didn’t vote are kicking themselves, and will participate next time.

And now for some good, maybe good. Trump is not going to make only bad decisions. There will be some policy changes that I am going to agree with. If the tax loopholes are actually closed, I will support that. If the trade deals are really negotiated more favorably to US, I will support that. If US stops getting involved militarily in Europe and elsewhere, I will support him, in spite of my deep disgust for him. Trump is neither a Democrat, nor a Republican. He hasn’t taken money from lobbyists. It’s possible that his decisions are done without any regard for the military-industrial complex, wall street, big pharma etc. Who knows what he will do, but it’s within the realm of possibility that some decisions would be very good for the common people.

And it could have been worse. It could have been Ted Cruz, who is a typical warmonger, in control of lobbyists and a champion of stone age thinking.

It’s still going to be bad, but it’s not the end of the world. If you are a progressive minded person, start working right now. Be prepared to oppose the regressive policies. Be prepared to protest. Be prepared to let your voice be heard by your representatives. It's our democracy too.

The four year clock has started.

Billions

Billions : Showtime Series

Aired on : Showtime (2016 - )

My Rating : 7 out of 10

Showtime, along with HBO, has become a channel to depend on for watching superb drama series. Whenever there is a new series on these channels, I am more than willing to invest my time watching it. Billions is their latest 2016 offering.

The main plot point of Billions is the clash between two ambitious male personalities. On one side is a billionaire hedge fund manager Bobby Axelrod (Damian Lewis). On the other side is a New York State Attorney, Chuck Rhoades (Paul Giamatti). Both are ambitious, egoistic and each has his own chip on his shoulder. Bobby has become rich from humble beginnings. He hasn’t forgotten his past, and who treated him well and who didn’t. He uses his money to make his point. Chuck is from a privileged background, and has a passionate hatred for rich who think they are above the law. He uses his legal authority to make his point.

Axelrod’s financial success has a troubling connection to 9/11. He is a ruthless manager. Supportive as well as manipulative, and vindictive towards those who betray him. He bends the laws, but is loyal to wife, and fiercely protective of his family. Chuck has the right ideas of bringing justice to financial criminals, but neither his motives not his means are all that pure. Both have egos bordering on petty. Their flaws and strengths, and their similarities and contrasts make the clash fascinating, otherwise it would turn into an uncomplicated good guy versus bad guy story.

Just to make sure that this is more than two alpha males fighting it out against each other, the story has two strong female characters, although not exactly in the lead roles. Bobby’s wife Lara (Malin Akerman) is a perfect cohort, tough and supportive. Chuck’s wife Wendy (Maggie Siff) is a very successful psychiatrist who makes a handsome salary because - get this - she works for Bobby Axelrod.

This is where the series pushes the envelope too far. Her motivational sessions with the employees are not that convincing. But the whole situation is unbelievable. The New York State Attorney who is eyeing for a public office in future has a wife that works for the same hedge fund billionaire who he trying to bring down. This kind of conflict of interest is unrealistic. The series tries to explain it away by informing us that, she was working there before Chuck’s rise. You decide, if you want to buy that as enough of a reason.

This is not a documentary, so some unrealisticness is not a problem, and the series does take such liberties. But this by far was the biggest one, which I think is a flaw. On the plus side, it does not take away the enjoyment, and in fact it allows the writers to add some interesting situations.

Overall. it’s not easy to relate to any of the characters, which I think is by design. It never gets intense as we are never emotionally vested into any character. We just watch with amusement and enjoy the fight.

A big part of that enjoyment is due to the strength of the performances, which are superb across the board. Not one bad acting performance. The pacing is fine, and the side stories of other characters don’t feel extraneous. The dialogues, on the other hand, feel overwritten on some occasions, and chuckle worthy on other occasions. 

This is an interesting series. I am looking forward to the second season. I don’t think this will win a Golden Globe or an Emmy, but if you watch it just for entertainment, I don’t think you will be disappointed. It’s correctly rated TV-MA and is not for children.

UPDATE : Season 2 review has been posted.