Category Archives: Housing

How the Economy Works

Read two great articles today. I’d like to synopsis the main points later, but merely wanted to post the articles for the time being

Timeline

https://imgur.com/gallery/xhYCQh4?fbclid=IwAR0ptSWHvLRIuzcI8iNydZrFVTpEh9WpQokYlKMQyMU9iblpWqjWjA_e-Hw

https://www.nytimes.com/interactive/2019/10/06/opinion/income-tax-rate-wealthy.html

The American Economy is Rigged

The American Economy Is Rigged – Scientific American

And what we can do about it

By Joseph E. Stiglitz on November 1, 2018
The American Economy Is Rigged
Credit: Andrea Ucini
Americans are used to thinking that their nation is special. In many ways, it is: the U.S. has by far the most Nobel Prize winners, the largest defense expenditures (almost equal to the next 10 or so countries put together) and the most billionaires (twice as many as China, the closest competitor). But some examples of American Exceptionalism should not make us proud. By most accounts, the U.S. has the highest level of economic inequality among developed countries. It has the world’s greatest per capita health expenditures yet the lowest life expectancy among comparable countries. It is also one of a few developed countries jostling for the dubious distinction of having the lowest measures of equality of opportunity.

The notion of the American Dream—that, unlike old Europe, we are a land of opportunity—is part of our essence. Yet the numbers say otherwise. The life prospects of a young American depend more on the income and education of his or her parents than in almost any other advanced country. When poor-boy-makes-good anecdotes get passed around in the media, that is precisely because such stories are so rare.

Things appear to be getting worse, partly as a result of forces, such as technology and globalization, that seem beyond our control, but most disturbingly because of those within our command. It is not the laws of nature that have led to this dire situation: it is the laws of humankind. Markets do not exist in a vacuum: they are shaped by rules and regulations, which can be designed to favor one group over another. President Donald Trump was right in saying that the system is rigged—by those in the inherited plutocracy of which he himself is a member. And he is making it much, much worse.

ADVERTISEMENT
America has long outdone others in its level of inequality, but in the past 40 years it has reached new heights. Whereas the income share of the top 0.1 percent has more than quadrupled and that of the top 1 percent has almost doubled, that of the bottom 90 percent has declined. Wages at the bottom, adjusted for inflation, are about the same as they were some 60 years ago! In fact, for those with a high school education or less, incomes have fallen over recent decades. Males have been particularly hard hit, as the U.S. has moved away from manufacturing industries into an economy based on services.

The Science of Inequality
Read more from this special report:
The Science of Inequality
DEATHS OF DESPAIR
Wealth is even less equally distributed, with just three Americans having as much as the bottom 50 percent—testimony to how much money there is at the top and how little there is at the bottom. Families in the bottom 50 percent hardly have the cash reserves to meet an emergency. Newspapers are replete with stories of those for whom the breakdown of a car or an illness starts a downward spiral from which they never recover.

In significant part because of high inequality, U.S. life expectancy, exceptionally low to begin with, is experiencing sustained declines. This in spite of the marvels of medical science, many advances of which occur right here in America and which are made readily available to the rich. Economist Ann Case and 2015 Nobel laureate in economics Angus Deaton describe one of the main causes of rising morbidity—the increase in alcoholism, drug overdoses and suicides—as “deaths of despair” by those who have given up hope.

Credit: Jen Christiansen; Sources: “The Fading American Dream: Trends in Absolute Income Mobility since 1940,” by Raj Chetty et al., in Science, Vol. 356; April 28, 2017 (child-parent wealth comparison); World Inequality database (90% versus 1% wealth trend data)
Defenders of America’s inequality have a pat explanation. They refer to the workings of a competitive market, where the laws of supply and demand determine wages, prices and even interest rates—a mechanical system, much like that describing the physical universe. Those with scarce assets or skills are amply rewarded, they argue, because of the larger contributions they make to the economy. What they get merely represents what they have contributed. Often they take out less than they contributed, so what is left over for the rest is that much more.

This fictional narrative may at one time have assuaged the guilt of those at the top and persuaded everyone else to accept this sorry state of affairs. Perhaps the defining moment exposing the lie was the 2008 financial crisis, when the bankers who brought the global economy to the brink of ruin with predatory lending, market manipulation and various other antisocial practices walked away with millions of dollars in bonuses just as millions of Americans lost their jobs and homes and tens of millions more worldwide suffered on their account. Virtually none of these bankers were ever held to account for their misdeeds.

ADVERTISEMENT
I became aware of the fantastical nature of this narrative as a schoolboy, when I thought of the wealth of the plantation owners, built on the backs of slaves. At the time of the Civil War, the market value of the slaves in the South was approximately half of the region’s total wealth, including the value of the land and the physical capital—the factories and equipment. The wealth of at least this part of this nation was not based on industry, innovation and commerce but rather on exploitation. Today we have replaced this open exploitation with more insidious forms, which have intensified since the Reagan-Thatcher revolution of the 1980s. This exploitation, I will argue, is largely to blame for the escalating inequality in the U.S.

After the New Deal of the 1930s, American inequality went into decline. By the 1950s inequality had receded to such an extent that another Nobel laureate in economics, Simon Kuznets, formulated what came to be called Kuznets’s law. In the early stages of development, as some parts of a country seize new opportunities, inequalities grow, he postulated; in the later stages, they shrink. The theory long fit the data—but then, around the early 1980s, the trend abruptly reversed.

EXPLAINING INEQUALITY
Economists have put forward a range of explanations for why inequality has in fact been increasing in many developed countries. Some argue that advances in technology have spurred the demand for skilled labor relative to unskilled labor, thereby depressing the wages of the latter. Yet that alone cannot explain why even skilled labor has done so poorly over the past two decades, why average wages have done so badly and why matters are so much worse in the U.S. than in other developed nations. Changes in technology are global and should affect all advanced economies in the same way. Other economists blame globalization itself, which has weakened the power of workers. Firms can and do move abroad unless demands for higher wages are curtailed. But again, globalization has been integral to all advanced economies. Why is its impact so much worse in the U.S.?

https://www.scientificamerican.com/article/the-american-economy-is-rigged/?print=true

The shift from a manufacturing to a service-based economy is partly to blame. At its extreme—a firm of one person—the service economy is a winner-takes-all system. A movie star makes millions, for example, whereas most actors make a pittance. Overall, wages are likely to be far more widely dispersed in a service economy than in one based on manufacturing, so the transition contributes to greater inequality. This fact does not explain, however, why the average wage has not improved for decades. Moreover, the shift to the service sector is happening in most other advanced countries: Why are matters so much worse in the U.S.?

Again, because services are often provided locally, firms have more market power: the ability to raise prices above what would prevail in a competitive market. A small town in rural America may have only one authorized Toyota repair shop, which virtually every Toyota owner is forced to patronize. The providers of these local services can raise prices over costs, increasing their profits and the share of income going to owners and managers. This, too, increases inequality. But again, why is U.S. inequality practically unique?

In his celebrated 2013 treatise Capital in the Twenty-First Century, French economist Thomas Piketty shifts the gaze to capitalists. He suggests that the few who own much of a country’s capital save so much that, given the stable and high return to capital (relative to the growth rate of the economy), their share of the national income has been increasing. His theory has, however, been questioned on many grounds. For instance, the savings rate of even the rich in the U.S. is so low, compared with the rich in other countries, that the increase in inequality should be lower here, not greater.

An alternative theory is far more consonant with the facts. Since the mid-1970s the rules of the economic game have been rewritten, both globally and nationally, in ways that advantage the rich and disadvantage the rest. And they have been rewritten further in this perverse direction in the U.S. than in other developed countries—even though the rules in the U.S. were already less favorable to workers. From this perspective, increasing inequality is a matter of choice: a consequence of our policies, laws and regulations.

In the U.S., the market power of large corporations, which was greater than in most other advanced countries to begin with, has increased even more than elsewhere. On the other hand, the market power of workers, which started out less than in most other advanced countries, has fallen further than elsewhere. This is not only because of the shift to a service-sector economy—it is because of the rigged rules of the game, rules set in a political system that is itself rigged through gerrymandering, voter suppression and the influence of money. A vicious spiral has formed: economic inequality translates into political inequality, which leads to rules that favor the wealthy, which in turn reinforces economic inequality.

FEEDBACK LOOP
Political scientists have documented the ways in which money influences politics in certain political systems, converting higher economic inequality into greater political inequality. Political inequality, in its turn, gives rise to more economic inequality as the rich use their political power to shape the rules of the game in ways that favor them—for instance, by softening antitrust laws and weakening unions. Using mathematical models, economists such as myself have shown that this two-way feedback loop between money and regulations leads to at least two stable points. If an economy starts out with lower inequality, the political system generates rules that sustain it, leading to one equilibrium situation. The American system is the other equilibrium—and will continue to be unless there is a democratic political awakening.

An account of how the rules have been shaped must begin with antitrust laws, first enacted 128 years ago in the U.S. to prevent the agglomeration of market power. Their enforcement has weakened—at a time when, if anything, the laws themselves should have been strengthened. Technological changes have concentrated market power in the hands of a few global players, in part because of so-called network effects: you are far more likely to join a particular social network or use a certain word processor if everyone you know is already using it. Once established, a firm such as Facebook or Microsoft is hard to dislodge. Moreover, fixed costs, such as that of developing a piece of software, have increased as compared with marginal costs—that of duplicating the software. A new entrant has to bear all these fixed costs up front, and if it does enter, the rich incumbent can respond by lowering prices drastically. The cost of making an additional e-book or photo-editing program is essentially zero.

ADVERTISEMENT
In short, entry is hard and risky, which gives established firms with deep war chests enormous power to crush competitors and ultimately raise prices. Making matters worse, U.S. firms have been innovative not only in the products they make but in thinking of ways to extend and amplify their market power. The European Commission has imposed fines of billions of dollars on Microsoft and Google and ordered them to stop their anticompetitive practices (such as Google privileging its own comparison shopping service). In the U.S., we have done too little to control concentrations of market power, so it is not a surprise that it has increased in many sectors.

Credit: Jen Christiansen; Sources: Economic Report of the President. January 2017; World Inequality database
Rigged rules also explain why the impact of globalization may have been worse in the U.S. A concerted attack on unions has almost halved the fraction of unionized workers in the nation, to about 11 percent. (In Scandinavia, it is roughly 70 percent.) Weaker unions provide workers less protection against the efforts of firms to drive down wages or worsen working conditions. Moreover, U.S. investment treaties such as the North Atlantic Free Trade Agreement—treaties that were sold as a way of preventing foreign countries from discriminating against American firms—also protect investors against a tightening of environmental and health regulations abroad. For instance, they enable corporations to sue nations in private international arbitration panels for passing laws that protect citizens and the environment but threaten the multinational company’s bottom line. Firms like these provisions, which enhance the credibility of a company’s threat to move abroad if workers do not temper their demands. In short, these investment agreements weaken U.S. workers’ bargaining power even further.

LIBERATED FINANCE
Many other changes to our norms, laws, rules and regulations have contributed to inequality. Weak corporate governance laws have allowed chief executives in the U.S. to compensate themselves 361 times more than the average worker, far more than in other developed countries. Financial liberalization—the stripping away of regulations designed to prevent the financial sector from imposing harms, such as the 2008 economic crisis, on the rest of society—has enabled the finance industry to grow in size and profitability and has increased its opportunities to exploit everyone else. Banks routinely indulge in practices that are legal but should not be, such as imposing usurious interest rates on borrowers or exorbitant fees on merchants for credit and debit cards and creating securities that are designed to fail. They also frequently do things that are illegal, including market manipulation and insider trading. In all of this, the financial sector has moved money away from ordinary Americans to rich bankers and the banks’ shareholders. This redistribution of wealth is an important contributor to American inequality.

Other means of so-called rent extraction—the withdrawal of income from the national pie that is incommensurate with societal contribution—abound. For example, a legal provision enacted in 2003 prohibited the government from negotiating drug prices for Medicare—a gift of some $50 billion a year or more to the pharmaceutical industry. Special favors, such as extractive industries’ obtaining public resources such as oil at below fair-market value or banks’ getting funds from the Federal Reserve at near-zero interest rates (which they relend at high interest rates), also amount to rent extraction. Further exacerbating inequality is favorable tax treatment for the rich. In the U.S., those at the top pay a smaller fraction of their income in taxes than those who are much poorer—a form of largesse that the Trump administration has just worsened with the 2017 tax bill.

Some economists have argued that we can lessen inequality only by giving up on growth and efficiency. But recent research, such as work done by Jonathan Ostry and others at the International Monetary Fund, suggests that economies with greater equality perform better, with higher growth, better average standards of living and greater stability. Inequality in the extremes observed in the U.S. and in the manner generated there actually damages the economy. The exploitation of market power and the variety of other distortions I have described, for instance, makes markets less efficient, leading to underproduction of valuable goods such as basic research and overproduction of others, such as exploitative financial products.

Credit: Jen Christiansen; Sources: World Inequality Report 2018. World Inequality Lab, 2017; Branko Milanovic
Moreover, because the rich typically spend a smaller fraction of their income on consumption than the poor, total or “aggregate” demand in countries with higher inequality is weaker. Societies could make up for this gap by increasing government spending—on infra-structure, education and health, for instance, all of which are investments necessary for long-term growth. But the politics of unequal societies typically puts the burden on monetary policy: interest rates are lowered to stimulate spending. Artificially low interest rates, especially if coupled with inadequate financial market regulation, can give rise to bubbles, which is what happened with the 2008 housing crisis.

It is no surprise that, on average, people living in unequal societies have less equality of opportunity: those at the bottom never get the education that would enable them to live up to their potential. This fact, in turn, exacerbates inequality while wasting the country’s most valuable resource: Americans themselves.

RESTORING JUSTICE
Morale is lower in unequal societies, especially when inequality is seen as unjust, and the feeling of being used or cheated leads to lower productivity. When those who run gambling casinos or bankers suffering from moral turpitude make a zillion times more than the scientists and inventors who brought us lasers, transistors and an understanding of DNA, it is clear that something is wrong. Then again, the children of the rich come to think of themselves as a class apart, entitled to their good fortune, and accordingly more likely to break the rules necessary for making society function. All of this contributes to a breakdown of trust, with its attendant impact on social cohesion and economic performance.

There is no magic bullet to remedy a problem as deep-rooted as America’s inequality. Its origins are largely political, so it is hard to imagine meaningful change without a concerted effort to take money out of politics—through, for instance, campaign finance reform. Blocking the revolving doors by which regulators and other government officials come from and return to the same industries they regulate and work with is also essential.

Credit: Jen Christiansen; Sources: Raising America’s Pay: Why It’s Our Central Economic Policy Challenge, by Josh Bivens et al. Economic Policy Institute, June 4, 2014; The State of Working America, by Lawrence Mishel, Josh Bivens, Elise Gould and Heidi Shierholz. 12th Edition. ILR Press, 2012
Beyond that, we need more progressive taxation and high-quality federally funded public education, including affordable access to universities for all, no ruinous loans required. We need modern competition laws to deal with the problems posed by 21st-century market power and stronger enforcement of the laws we do have. We need labor laws that protect workers and their rights to unionize. We need corporate governance laws that curb exorbitant salaries bestowed on chief executives, and we need stronger financial regulations that will prevent banks from engaging in the exploitative practices that have become their hallmark. We need better enforcement of antidiscrimination laws: it is unconscionable that women and minorities get paid a mere fraction of what their white male counterparts receive. We also need more sensible inheritance laws that will reduce the intergenerational transmission of advantage and disadvantage.

The basic perquisites of a middle-class life, including a secure old age, are no longer attainable for most Americans. We need to guarantee access to health care. We need to strengthen and reform retirement programs, which have put an increasing burden of risk management on workers (who are expected to manage their portfolios to guard simultaneously against the risks of inflation and market collapse) and opened them up to exploitation by our financial sector (which sells them products designed to maximize bank fees rather than retirement security). Our mortgage system was our Achilles’ heel, and we have not really fixed it. With such a large fraction of Americans living in cities, we have to have urban housing policies that ensure affordable housing for all.

It is a long agenda—but a doable one. When skeptics say it is nice but not affordable, I reply: We cannot afford to not do these things. We are already paying a high price for inequality, but it is just a down payment on what we will have to pay if we do not do something—and quickly. It is not just our economy that is at stake; we are risking our democracy.

As more of our citizens come to understand why the fruits of economic progress have been so unequally shared, there is a real danger that they will become open to a demagogue blaming the country’s problems on others and making false promises of rectifying “a rigged system.” We are already experiencing a foretaste of what might happen. It could get much worse.

Joseph E. Stiglitz

Joseph E. Stiglitz is a University Professor at Columbia University and Chief Economist at the Roosevelt Institute. He received the Nobel prize in economics in 2001. Stiglitz chaired the Council of Economic Advisers from 1995–1997, during the Clinton administration, and served as the chief economist and senior vice president of the World Bank from 1997–2000. He chaired the United Nations commission on reforms of the international financial system in 2008–2009. His latest authored book is Globalization and Its Discontents Revisited (2017).

Credit: Nick Higgins

[AOC Thinks] Concentrated Wealth is Incompatible With Democracy. So Did Our Founders

AOC Thinks Concentrated Wealth Is Incompatible With Democracy. So Did Our Founders.

By Eric Levitz@EricLevitz

No Paine, no gain. Photo: White House Collection; US House of Representatives; National Portrait Gallery
In 1835, Alexis de Tocqueville produced one of the earliest accounts of the American dream. In his famous study of the Jacksonian U.S., the Frenchman wrote that Americans possessed “the charm of anticipated success” — a ubiquitous optimism that he attributed to our country’s democratic character, and to the “general equality of condition” that prevailed among its “people.”

On Wednesday night, Sean Hannity took de Tocqueville to task. In the Fox News’ host’s telling, general economic equality is not a precondition for the American dream, but rather, an insurmountable obstacle to it — because the American dream is (apparently) to earn more than $10 million year without having to pay a top marginal tax rate higher than 37 percent.

Of course, Hannity did not actually frame his argument as a rebuke of de Tocqueville. His true target was Alexandria Ocasio-Cortez.

After popularizing the idea of a 70 percent top marginal tax rate earlier this month, the freshman congresswoman recently suggested that the mere existence of billionaires was both immoral, and a threat to American democracy. “I do think that a system that allows billionaires to exist when there are parts of Alabama where people are still getting ringworm because they don’t have access to public health is wrong,” Ocasio-Cortez told the writer Ta-Nehisi Coates, during an interview on Martin Luther King Day. One day later, the congresswoman approvingly quoted an op-ed by the economists Gabriel Zucman and Emmanuel Saez, which argued that the purpose of high taxes on the wealthy wasn’t merely to generate revenue, but rather, to safeguard “democracy against oligarchy.”

Hannity’s not buying it. The Fox News host informed his audience Wednesday that Ocasio-Cortez had “called the American dream immoral,” and that she wants to “empower the government to confiscate” said dream. “Better hide your nice things,” Hannity advised his audience (whom he ostensibly believes to be composed primarily of billionaires), “because here come the excess police.”

Hannity was hardly alone in deriding AOC’s antipathy for billionaires as fundamentally un-American. But in reality, there’s nothing foreign or communistic about the idea that concentrated wealth is incompatible with democracy, or all-too compatible with mass poverty. Republicans might call such notions radical. But many of our republic’s founders would have called them common sense.

Compare AOC’s first argument — that the simultaneous existence of billionaires and poverty is immoral, and thus justifies steeply progressive taxation — with Thomas Jefferson’s reflections in 1785. During a visit to the French countryside, Jefferson found himself scandalized by “the condition of the labouring poor.” In a letter to James Madison, Jefferson wrote that the extremity of European inequality was not only morally suspect, but economically inefficient. Aristocrats had grown so wealthy, they were happy to leave their lands uncultivated, even as masses of idle workers were eager to improve it. Thus, these proto-billionaires undermined both the peasants’ ability to transcend mere subsistence, and their society’s capacity to develop economically:

[T]he solitude of my walk led me into a train of reflections on that unequal division of property which occasions the numberless instances of wretchedness which I had observed in this country and is to be observed all over Europe. The property of this country is absolutely concentered in a very few hands…I asked myself what could be the reason that so many should be permitted to beg who are willing to work, in a country where there is a very considerable proportion of uncultivated lands? These lands are kept idle mostly for the aske of game. It should seem then that it must be because of the enormous wealth of the proprietors which places them above attention to the increase of their revenues by permitting these lands to be laboured.

Here is how Jefferson proposes to address the obscene coexistence of concentrated wealth and underemployed workers:

I am conscious that an equal division of property is impracticable. But the consequences of this enormous inequality producing so much misery to the bulk of mankind, legislators cannot invent too many devices for subdividing property, only taking care to let their subdivisions go hand in hand with the natural affections of the human mind. The descent of property of every kind therefore to all the children, or to all the brothers and sisters, or other relations in equal degree is a politic measure, and a practicable one. Another means of silently lessening the inequality of property is to exempt all from taxation below a certain point, and to tax the higher portions of property in geometrical progression as they rise. Whenever there is in any country, uncultivated lands and unemployed poor, it is clear that the laws of property have been so far extended as to violate natural right…It is too soon yet in our country to say that every man who cannot find employment but who can find uncultivated land, shall be at liberty to cultivate it, paying a moderate rent. But it is not too soon to provide by every possible means that as few as possible shall be without a little portion of land. The small landholders are the most precious part of a state. [Emphasis mine.]

If Ocasio-Cortez’s views are un-American, then surely these words from our third president’s are, as well.

To be sure, Jefferson’s views on the propriety of wealth redistribution were hardly consistent. And, of course, the slave owner was never concerned with minimizing the number of landless African-Americans or women in the United States. What’s more, the bulk of America’s founders regarded wealth redistribution as a species of majoritarian tyranny, and designed the Constitution to guard against such despotism.

My point here isn’t to suggest that AOC is channeling the sacred wisdom of our republic’s founding racists. Rather, it’s that she’s channeling one deeply rooted strain of American thought on economic morality. And while that strain might have been marginal among the leaders of the American Revolution, it was pervasive among its foot soldiers (there’s a reason the leading propagandist of the war effort, Thomas Paine, was one of the earliest champions of an American welfare state).

Regardless, Ocasio-Cortez’s second argument against the existence of billionaires — that concentrated wealth is incompatible with genuine democracy — was something close to conventional wisdom among the founders (including those who opposed democracy).

America’s first political theorists took these truths to be self-evident: that a person could not exercise political liberty if he did not possess a modicum of economic autonomy, and that disparities in wealth inevitably produced disparities of political power.

The notion that political freedom has a material basis did not originate with Karl Marx and the creed of Communism; it was a core idea of the 17th-century British political theorist James Harrington, and his formulation of classical republicanism. A man who does not own the means of his own reproduction can never exercise political freedom, Harrington argued, because “the man that cannot live upon his own must be servant.” Likewise, the man of immense wealth — whose fortune consigns great masses of men to servitude — is inevitably a kind of tyrant. After all, “where there is inequality of estates, there must be inequality of power, and where there is inequality of power, there can be no commonwealth.”

These premises deeply informed the American founders’ conception of republican liberty. The Jeffersonian ideal of a yeoman’s republic derived from the conviction that only independent landowners were politically free — and only a (very) rough equality in the distribution of land could preserve such freedom. Even a consummate elitist like Alexander Hamilton couldn’t help but echo Harringtonian thinking, writing in the Federalist Papers, “A power over man’s subsistence amounts to a power over his will.”

Critically, relatively few of the founders saw these premises in a progressive light. To many 18th-century American elites, the fact that the propertyless lacked the capacity to exercise genuine political freedom was not an argument for giving them property, but rather, for denying them the franchise. Similarly, the notion that true democracy couldn’t coexist with wealth inequality struck many leaders of the early republic as an argument against democracy.

“Power and property may be seperated for a time, by force or fraud — but divorced never, ” Benjamin Leigh, a conservative legislator in Virginia’s House of Delegates, argued at that state’s Constitutional Convention in 1830. “For, so soon as the pang of separation is felt … property will purchase power, or power will take property.” Being a man of property, Leigh concluded that the poor should therefore be denied political rights, saying, “it does not follow that, because all men are born equal … all men may rightly claim, in an established society, equal political powers.”

Thus, Ocasio-Cortez’s belief in the moral necessity of mass democracy (and women’s suffrage, and the abolition of slavery) would have struck many a Founding Father as radical. But her insistence that true democracy is incompatible with America’s present distribution of property — in which the richest 0.1 percent of Americans command as much wealth as the poorest 90 percent — would have struck Jefferson & Co. as tautological. And a large body of political science research suggests that their shared intuition is correct.

All of which is to say: If the right to self-government is an inextricable component of the American dream, then it isn’t AOC who regards that dream as immoral — it’s Sean Hannity, and every other multimillionaire who believes that legislators should not invent “many devices for subdividing property.”

Property

I work because I don’t own property (I have a mortgage). If I had property, depending on it’s value, I could trade some of it to acquire goods.

Property is the proverbial carrot on the stick in American life. You want property where it’s appealing. By keeping us from acquiring it and focusing on consumer goods, keeps us addicted to the american economy.

So you work and pay rent to live in an appealing place. If you don’t own property, you don’t have any option not to pay rent. By keeping the acquisition of property artificially high with barriers to entry (such as 10% or what have you, which is arguably creating more access, but you get my point when I mention how VA home loans and FHA loans allow subsidies to qualified buyers) forces people into categories of more and less privileged when it comes to gaining access to it so begin the rat race of competition where the government can now offer subsidized school loans as a type of indentured servitude to the american economy all in the hopes of acquiring that dangling carrot on the stick. Property.

That last piece. It’s value is artificially inflated. It’s mainly hubris because no one really acquires it (some do, but that’s the point, it’s like a pyramid scheme). We’re all kind of floating on varying layers of acquisition of it with our rents, mortgages, and (or not) equity.

By keeping it at bay, it remains a proverbial carrot on the stick while most people don’t even realize it’s impact on our daily lives of the rat race. By keeping property out of our reach, it forces us to work.

Categorical Up/Down Housing Yield mapped to scatterplot matrix

I did it again, I accomplished a visual display of data that helps me identify how trends affect the outcome variable. I have graphed here the up/down response variable (was yield positive? then yellow, negative, green (I know, not intuitive)).

I can use this graph tukey 7 number summary of response variables into “profiles”. I already had another Eureka moment earlier with the way that knn distances can be potentially mapped (and I will return with a graph of that later) to return (see prior post)

Bug report to actually graph this

I’m actually at the point in my code where I want to start documenting change requests.

Alternative KNN Distance Algorithm

Continuing on this post

& this post

Euclidean Distance is Sqrt((x – u)^2 + …)

for each x to each predictor x (u)

However,  I derived the idea to use “mew” for u instead of the predictor u and I’m glad I did.  The inference that distance is squared is used to measure distance as a non negative value, which makes mapping it to response value difficult.  So I revamped the way I calculated my distance.  It’s basically a normalized set of columns and each row is summed up and then compared with the response.

If you note in distance2 (which is normalized, distance3 is not normalized).  I have an almost linear relationship.  This is with no coefficients mapped.  Simply yields.  I KNOW these factors work well, but I haven’t actually done any modelling.  I have another idea to square the distance like I have, but to preserve signs (and hopefully preserve sign at the tail end when it’s square rooted!).  I guess I shouldn’t care so much considering all the variables are already normalized (distance2 has normalized factors using method here

That’s what’s so great about this.  I now have a distance to housing yield formula.

Alt KNN Housing Yield Analysis

This is really exciting, I applied my alternative knn formula (which completely ignores the response variable), and found the highest distance value and it’s the one outlier I’ve been noticing in all my studentized deleted residual plots and I was able to quickly identify why because on 3/31/2012 the quarterly yield for TB3MS jumped 7 standard deviations (over 30 years!), also one of my highest yields. I can use this matrix in a similar setup to knn to make predictions.  I’m going to pull up the values I had for my unreported quarter (9 something or other) and do exactly that.

Alt KNN

 

Prediction

Unfortunately, the value is at the edge of my ranked distances, it has the lowest distance.

BoxPlots

Housing Model 2

DBScan (Density) of Housing Model

https://rdrr.io/cran/dbscan/man/kNNdist.html

http://www.sthda.com/english/wiki/print.php?id=246

Prediction for 12/31/2018

Changes from prior model

Dropped NYXRSA

Adj R^2 .797

Estimated @ 95% confidence .004645 +/- (.0083*2)

Max Absolute Residual: .024 (2.4%)

Schiller’s Index Yield is expected to increased by .4%

Expected Value (not a predicted value)

with a 95% confidence interval at +/- 1.66% (not prediction interval)

or

-1.26 to 2.06 %

(Intercept) -0.010161
xYield_CASTHPI 0.355302
xYield_TB3MS 0.006162
xYield_MSPNHSUS 0.124203
xYield_MVLOAS 0.249076
Q1 0.02504
Q2 0.017209

Adj R^2

Best CV Housing Model 7 Factors

I was inspired/challenged to build a CV model of next quarter’s housing, so I reran it and here is the end result. My R^2 is pretty good @ .818 w 7 factors. I checked a few model assumptions, one studentized residual is almost 4 (2004-03-31), I’m still a little shakey on what rules to process outlier’s, I don’t feel comfortable pulling them. I’m not sure if I’m reading the std error of the estimate is less than 1% (the values represent % change, so std error is .007), the max residual is .04, which tells me the max error I received was 4%.

From file regression_analysis.R

Commit: https://github.com/thistleknot/FredAPIR/commit/da48081571a5bf7d7e248860832543010becf053

E = -0.009797334 + xYield_NYXRSA*0.229614126 + xYield_CASTHPI*0.262771376 + xYield_TB3MS*0.006684356 + xYield_MSPNHSUS*0.103737365 + xYield_MVLOAS*0.200071463 + Q1*0.023881913 + Q2*0.017111071

xYield_NYXRSA
xYield_CASTHPI
xYield_TB3MS
xYield_MSPNHSUS
xYield_CPALTT01USQ657N
xYield_MVLOAS
Q1
Q2

Machine Learning (Stepwise) Yield Housing Model

Output is from derived factors using step-wise regression of the following variables over 164 months (jan 31, 1990)

“CPIAUCSL”,”UNRATE”,”MEHOINUSA672N”,”CIVPART”,”INTDSRUSM193N”,”FEDFUNDS”,”GDPC1″,”A191RL1Q225SBEA”,”CSUSHPINSA”,”SP500″,”DCOILWTICO”,”DFII10″,”DFF”,”SPCS20RSA”,”GDPDEF”,”PAYEMS”,”CES0500000003″,”INDPRO”,”M2V”,”UMCSENT”,”PCE”,”STLFSI”,”BASE”,”PSAVERT”,”M2″,”M1″,”T10Y2Y”,”DGS10″,”BAMLH0A0HYM2″,”TB3MS”,”T10Y3M”,”T10YIE”,”TEDRATE”,”GFDEGDQ188S”,”USSLIND”,”T5YIFR”,”DGS2″,”DGS1″,”BAMLC0A4CBBB”,”GS10″,”GFDEBTN”,”DGS5″,”TTLHH”,”EMRATIO”,”GOLDAMGBD228NLBM”,”POPTOTUSA647NWDB”,

Using Yield

                            OLS Regression Results                            
==============================================================================
Dep. Variable:             CSUSHPINSA   R-squared:                       0.953
Model:                            OLS   Adj. R-squared:                  0.946
Method:                 Least Squares   F-statistic:                     146.5
Date:                Sun, 21 Oct 2018   Prob (F-statistic):           1.23e-85
Time:                        01:16:03   Log-Likelihood:                 860.71
No. Observations:                 165   AIC:                            -1681.
Df Residuals:                     145   BIC:                            -1619.
Df Model:                          20                                         
Covariance Type:            nonrobust                                         
===============================================================================
                  coef    std err          t      P>|t|      [0.025      0.975]
-------------------------------------------------------------------------------
DGS1        -6.362e-05      0.000     -0.270      0.788      -0.001       0.000
PSAVERT         0.0003      0.000      1.197      0.233      -0.000       0.001
GDPC1        -2.58e-06      3e-06     -0.858      0.392   -8.52e-06    3.36e-06
GFDEBTN       2.76e-09   3.69e-09      0.748      0.456   -4.53e-09    1.01e-08
GFDEGDQ188S    -0.0004      0.000     -0.903      0.368      -0.001       0.000
CIVPART         0.0011      0.001      1.967      0.051   -5.25e-06       0.002
M2V            -0.0080      0.007     -1.182      0.239      -0.021       0.005
BASE         2.954e-05   1.16e-05      2.540      0.012    6.56e-06    5.25e-05
DFII10         -0.0069      0.002     -3.667      0.000      -0.011      -0.003
DGS1            0.0096      0.003      3.665      0.000       0.004       0.015
PSAVERT        -0.0019      0.001     -1.394      0.165      -0.005       0.001
GDPC1           0.0511      0.333      0.153      0.878      -0.608       0.710
GFDEBTN        -0.1232      0.362     -0.340      0.734      -0.839       0.592
GFDEGDQ188S    -0.0373      0.359     -0.104      0.917      -0.746       0.671
CIVPART        -0.0421      0.062     -0.679      0.498      -0.165       0.080
M2V            -0.0758      0.089     -0.856      0.394      -0.251       0.099
BASE           -0.0175      0.019     -0.917      0.361      -0.055       0.020
DFII10         -0.0008      0.004     -0.221      0.826      -0.008       0.006
CSUSHPINSA     -0.0001    5.4e-05     -2.760      0.007      -0.000   -4.23e-05
CSUSHPINSA      0.7441      0.043     17.342      0.000       0.659       0.829
==============================================================================
Omnibus:                        5.887   Durbin-Watson:                   1.200
Prob(Omnibus):                  0.053   Jarque-Bera (JB):                5.891
Skew:                           0.463   Prob(JB):                       0.0526
Kurtosis:                       2.969   Cond. No.                     3.01e+10
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 3.01e+10. This might indicate that there are
strong multicollinearity or other numerical problems.
Variables extended to interactions prior record (i.e. *3 k)
Out[163]:
OLS Regression Results
Dep. Variable: CSUSHPINSA R-squared: 0.894
Model: OLS Adj. R-squared: 0.883
Method: Least Squares F-statistic: 78.40
Date: Sun, 21 Oct 2018 Prob (F-statistic): 8.67e-64
Time: 01:16:03 Log-Likelihood: 860.62
No. Observations: 165 AIC: -1687.
Df Residuals: 148 BIC: -1634.
Df Model: 16
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
DGS1 -0.0002 0.001 -0.241 0.810 -0.002 0.002
PSAVERT -0.0002 0.001 -0.388 0.699 -0.001 0.001
GDPC1 -4.865e-05 1.73e-05 -2.807 0.006 -8.29e-05 -1.44e-05
GFDEBTN 1.156e-08 1.58e-08 0.734 0.464 -1.96e-08 4.27e-08
GFDEGDQ188S 0.0018 0.002 0.868 0.387 -0.002 0.006
CIVPART 0.0072 0.002 3.596 0.000 0.003 0.011
M2V 0.0014 0.001 2.845 0.005 0.000 0.002
BASE 0.0001 7.34e-05 1.681 0.095 -2.16e-05 0.000
DFII10 -0.0025 0.002 -1.607 0.110 -0.006 0.001
DGS1 -0.0025 0.001 -3.079 0.002 -0.004 -0.001
PSAVERT -0.0002 0.001 -0.424 0.672 -0.001 0.001
GDPC1 -5.156e-05 2.16e-05 -2.382 0.018 -9.43e-05 -8.79e-06
GFDEBTN 4.924e-08 2.08e-08 2.362 0.019 8.04e-09 9.04e-08
GFDEGDQ188S 0.0022 0.001 2.279 0.024 0.000 0.004
CIVPART 0.0074 0.002 3.736 0.000 0.003 0.011
M2V 0.0013 0.000 2.757 0.007 0.000 0.002
BASE 0.0001 8.21e-05 1.285 0.201 -5.67e-05 0.000
DFII10 0.0021 0.002 1.346 0.180 -0.001 0.005
DGS1 0.0002 0.000 1.616 0.108 -4.11e-05 0.000
PSAVERT 5.393e-05 7.42e-05 0.727 0.469 -9.27e-05 0.000
GDPC1 3.248e-09 1.26e-09 2.575 0.011 7.55e-10 5.74e-09
GFDEBTN -2.354e-15 1.91e-15 -1.230 0.221 -6.14e-15 1.43e-15
GFDEGDQ188S -6.816e-05 3.19e-05 -2.137 0.034 -0.000 -5.12e-06
CIVPART -0.0001 2.72e-05 -3.943 0.000 -0.000 -5.34e-05
M2V 0.0046 0.002 2.483 0.014 0.001 0.008
BASE -1.37e-07 1.23e-07 -1.116 0.266 -3.8e-07 1.06e-07
DFII10 -0.0009 0.000 -1.942 0.054 -0.002 1.55e-05
CSUSHPINSA 0.0068 0.001 13.112 0.000 0.006 0.008
CSUSHPINSA -0.0083 0.001 -15.338 0.000 -0.009 -0.007
CSUSHPINSA 3.208e-06 2.47e-06 1.300 0.196 -1.67e-06 8.08e-06
Omnibus: 2.484 Durbin-Watson: 1.086
Prob(Omnibus): 0.289 Jarque-Bera (JB): 2.124
Skew: 0.163 Prob(JB): 0.346
Kurtosis: 2.550 Cond. No. 2.00e+16

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 2e+16. This might indicate that there are
strong multicollinearity or other numerical problems.

In [ ]:

Python Housing Model with SciKit

A Professional in the field named Emma Muhleman recommended that I use scikit.  I’ve seen it before but never paid attention to anything Python because I wanted to focus on R.  Initially I dived in with statsmodels rather than scikit, but now I have my hooks into scikit with linear regression modelling!

Remote Dev Environment

The underlying model I’m using has 9 factors that all test significant for housing and had a .999 Adj R^2 when predicting nominal (not necessarily future, I went straight to yield because I knew I wouldn’t have to worry about min/max boundaries as much with yield).

Add on analysis

I derive the yield of these factors using a simple offset formula.  So each predictor variable is extended by one.  This in turn results in a directional measurement between nominal values (between records) as a %, assuming the stock market is on an up or downward trend, we should expect returns to be similar to an index (maybe it’s symmetrical, idk).

SciKit

Initial article that got me into scikit linear regression modeling, I copied this url into my code.

https://towardsdatascience.com/train-test-split-and-cross-validation-in-python-80b61beca4b6

With the scikit, concerning my method for adding yield, I figured it didn’t matter what sequential sets of values I fed into the Multiple Regression equation anymore.  Because I effectively decoupled the relevant time series information.  I basically got the before and after record with yield.  It’s probably overkill to derive the yield, the nominal may have been adequate.  But I certainly needed the yield for the predicted variable (Y)

Here’s the scripts inspired

https://github.com/thistleknot/Python-Scripts

Future Direction

It wasn’t overkill to use yields.  Apparently this is known as nth order differencing in data science and I was correct to implement it, but incorrect to assume it was only a distance of 1.  Generally 1 is all that is needed in business applications, but the proper way to determine the proper lag is measuring autocorrelation (and looking at a correlogram).  However, I’m unsure how to modify lag for multiple variables using autocorrelation, so maybe simply setting it to 1 was adequate.  I’ve read a paper that values less than 1 are also beneficial to maintaining some of the information (such as .7, which I’m not sure how one would implement that without interpolation of data).  Actually I do remember.  Each variable has it’s own independent autocorrelation lag.

Score: 0.8940913536570876
                            OLS Regression Results                            
==============================================================================
Dep. Variable:             CSUSHPINSA   R-squared:                       0.945
Model:                            OLS   Adj. R-squared:                  0.938
Method:                 Least Squares   F-statistic:                     125.6
Date:                Tue, 16 Oct 2018   Prob (F-statistic):           4.50e-81
Time:                        18:52:42   Log-Likelihood:                 848.68
No. Observations:                 165   AIC:                            -1657.
Df Residuals:                     145   BIC:                            -1595.
Df Model:                          20                                         
Covariance Type:            nonrobust                                         
====================================================================================
                       coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------------
CPIAUCSL         -3.546e-05      0.000     -0.227      0.820      -0.000       0.000
PSAVERT              0.0003      0.000      1.334      0.184      -0.000       0.001
GDPC1            -6.812e-07   1.73e-06     -0.395      0.694   -4.09e-06    2.73e-06
DGS10               -0.0002      0.000     -0.723      0.471      -0.001       0.000
UMCSENT           8.211e-05   3.61e-05      2.274      0.024    1.07e-05       0.000
EMRATIO              0.0007      0.001      1.256      0.211      -0.000       0.002
POPTOTUSA647NWDB -3.733e-10   2.25e-10     -1.660      0.099   -8.18e-10    7.13e-11
TTLHH             7.295e-07   4.78e-07      1.526      0.129   -2.15e-07    1.67e-06
MEHOINUSA672N      4.76e-09   3.25e-07      0.015      0.988   -6.38e-07    6.48e-07
CPIAUCSL             0.0407      0.082      0.500      0.618      -0.120       0.202
PSAVERT             -0.0007      0.001     -0.507      0.613      -0.004       0.002
GDPC1                0.2813      0.097      2.912      0.004       0.090       0.472
DGS10                0.0105      0.003      3.418      0.001       0.004       0.017
UMCSENT             -0.0018      0.003     -0.564      0.573      -0.008       0.005
EMRATIO             -0.0193      0.056     -0.346      0.730      -0.130       0.091
POPTOTUSA647NWDB   -11.8906      3.255     -3.653      0.000     -18.324      -5.457
TTLHH               -0.4042      0.296     -1.365      0.174      -0.989       0.181
MEHOINUSA672N       -0.1396      0.161     -0.865      0.388      -0.459       0.179
CSUSHPINSA         3.58e-05   4.92e-05      0.727      0.468   -6.15e-05       0.000
CSUSHPINSA           0.7776      0.046     17.075      0.000       0.688       0.868
==============================================================================
Omnibus:                        2.936   Durbin-Watson:                   1.001
Prob(Omnibus):                  0.230   Jarque-Bera (JB):                2.467
Skew:                           0.253   Prob(JB):                        0.291
Kurtosis:                       3.321   Cond. No.                     7.64e+12
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 7.64e+12. This might indicate that there are
strong multicollinearity or other numerical problems.

Confusion Matrix and Lift Chart

Predicted vs Actual
            0  CSUSHPINSA
165  0.007443         NaN
166  0.008103    0.007467
167  0.007900    0.009214
168  0.008979    0.011215
169  0.010078    0.016365
170  0.013900    0.014692
171  0.011516    0.015222
172  0.011742    0.014575
173  0.012325    0.012110
174  0.011195    0.010494
175  0.008826    0.009601
176  0.007586    0.006435
177  0.004642    0.005163
178  0.004296    0.002415
179  0.002599    0.004003
180  0.004402    0.003711
181  0.004060    0.006876
182  0.005702    0.004908
183  0.004757    0.003986
184  0.002614    0.000922
185  0.000465    0.000347
186 -0.000181   -0.001094
187 -0.001341   -0.001122
188 -0.001080   -0.000787
189 -0.000091   -0.002304
190 -0.001454   -0.002189
191 -0.001278   -0.002789
192 -0.001165   -0.001352
193  0.001183   -0.001518
194 -0.000264   -0.000340
..        ...         ...
297 -0.003397    0.000577
298 -0.000941   -0.000126
299 -0.002712   -0.000474
300 -0.003365    0.001376
301 -0.001638    0.007522
302  0.004968    0.010720
303  0.006377    0.010393
304  0.006372    0.008790
305  0.004347    0.006137
306  0.001748    0.003560
307  0.000945    0.001649
308 -0.000200    0.000435
309 -0.001317    0.001227
310  0.001100    0.001020
311  0.000510    0.001376
312 -0.000968    0.002078
313  0.007335    0.008125
314  0.012493    0.010738
315  0.013645    0.010597
316  0.014212    0.009097
317  0.012367    0.006536
318  0.011464    0.004372
319  0.008880    0.002513
320  0.007846    0.001441
321  0.007916    0.002002
322  0.007413    0.002095
323  0.007389    0.001387
324  0.007532    0.004078
325  0.010615    0.008595
326  0.013267    0.010397

[162 rows x 2 columns]