Just another WordPress site

Blog

  • We Did Not “Lose” 1.2 Million People (Updated)

    UPDATE: Nate Silver pointed me to the BLS extrapolation of their population adjustment data (Table C at the bottom of this release). Long story short: the 1.2 million change in “Not in the Labor Force” is due entirely to the population adjustment and is not (as I assumed when writing this post) a straight across-the-board proportional increase. From December-January, we actually saw a “Not in the Labor Force” decrease which would mean people returning to the workforce. This is awesome. I’ll leave this post here as a testament to my own ignorance.

    BLS data came out today and, if you follow me on Twitter, you’ll know that’s kind of “my thing”. I tried to explain this in tweet form, but it requires a bigger canvas.

    There was some noise made today that we say “1.2 million people drop out of the workforce”. Zero Hedge and Iowahawk, both of whom are usually really good with numbers, made this claim.

    And I want to explain why that isn\’t so.

    First, let’s look at where that number comes from. In the BLS Employment tables, there is a stat called “Not In The Work Force”. That number rose from 86.7 million in December 2011 to 87.9 million in January 2012, a rise of 1.2 million in one month.

    Usually, when I talk about “people dropping out of the labor force”, I’m talking about the actual stat “Civilian Labor Force Level” decreasing as a raw number. This isn’t what happened. In fact, the Labor Force jumped half a million between December and January (the third biggest jump since the recession began).

    So… the labor force rose dramatically, but “not in the labor force” also rose dramatically. Why is that?

    The answer lies in the population. Normally population increases are a super-boring statistic in the job report. They always go up about the same amount, 150K-200K per month. However, the BLS does an annual adjustment every January. The last two adjustments have been downward, but this January, the adjustment was a huge upward one.


    That\’s a population adjustment of 1.7 million people. That’s big. In fact, that\’s second biggest population adjustment in BLS data ever.

    So when the population is adjusted, everything else in the data set gets adjusted too. The Labor Force goes up, Employment goes up, and the Not in the Labor Force goes up.

    Now, the concerning thing about this adjustment, and the only mote in an otherwise spotless jobs report, is the distribution of this adjustment. We have a labor force participation rate of 63.7% (which is very, very low). If we add a hundred people, ideally we’d like to see at least 64 of them added join the workforce (to keep that participation rate up). We want to see the labor force grow at least on the level of the population.

    Instead, what we got was this:

    The BLS added 1.6 million people to the population number but only 30% of those were added to the Labor Force. That\’s upside down from what we should expect in a normal adjustment.

    Now… what does this mean?

    Here is my theory based on how I understand the stats work.

    At the end of 2011, the BLS re-calculates their population numbers and says “Whoa… we’ve way undercounted the population, so let’s adjust that number.” They adjust that number up and all the other numbers with it go up too. So what happened was that the “Not in the labor force” count did go up, but the population adjustment amplified the increase. And the Labor Force probably went down, but the population adjustment made it look like it’s going up.

    I did some rough calculations (adjusting the December population upwards so that it more closely matches the January population) and, if we smooth the population adjustment, the jobs report looks less rosy. It looks like we lost another 450K from the labor force, which accounts for nearly all of the drop in unemployment.

    The complicated nature of these numbers means that this observation won’t get very much traction (and, honestly, I’m not 100% convinced I’ve accounted for everything so maybe it shouldn’t) but it fits into the overall observation that employment increases are just not keeping up with population increases. I wouldn’t say this is alarming, but it is concerning.

  • February 2012 BLS Data in Excel (CSV) Format

    Trying to get better at posting these soon after they are released.

    BLS A Tables (Labor Force, Employment Rates, Discouraged Workers, etc)

    BLS B Tables (Total Non-Farm Payrolls, Private Sector Jobs, Government Jobs, etc)

    For more information on this ongoing project of mine.

  • BLS Data to Excel Format & Source Code

    Last month, I published BLS Data in Excel format. This can be helpful for anyone who ever wants to really dig into the data but doesn\’t have the time to pull data out of the atrocious BLS data tables.

    I\’m going to try to make this something of a monthly thing, putting these data files on my website as soon as I can convert them so that the latest BLS data is always available in a helpful format.

    Additionally, I\’ve added the files for employment by metro. It\’s only up till September, 2011, but it\’s still some super cool data.

    January 2012 BLS Files

    A Tables (Employment/Unemployment)

    B Tables (Employment By Industry)

    State Employment/Unemployment

    State Employment By Industry

    Metro Employment up to September 2011

    Brief Interruption To Beg

    This took a not-insignificant amount of time and if you use it in anything resembling a professional capacity, I\’d really appreciate a beer as a way of saying thank you.

    \"\"

     

    BLS-To-Excel Application

    For those of you who are a little more interested in the data and willing to follow a lot of directions, I\’ve decided to publish the program I use for this so that you\’re not reliant on me to publish this every month. I do mostly Microsoft development, so you\’ll need Windows to run the project

    BLS Data To Excel Setup

    I\’ve also loaded the project to github so you can go download the source code and make it better.

    BLS-Data-To-CSV on github

    The code is a disaster in a large part because the BLS data is something of a disaster. However, the app itself contains some helpful tutorials on how to get the data and make everything work.

    It looks awful. But if you follow the directions, it works.

    This will never be a professional application, but I\’ll update it as I can. If you happen to have any talent in design, my \”thing\” is translating designs to reality. So if you want to send me even a screenshot of how you think this app should work, I\’m happy to incorporate that into the next version.

  • BLS Data in Excel Format

    If you\’ve ever tried to get the data out of the Bureau of Labor Statistics (BLS) you know that can be kind of a pain in the butt. You usually have to go through the wizards to get the data and then it only gives you one kind of data per table and you have to do a lot of tedious work to get that data into a format that is actually useful for additional work.

    I finally got tired of doing this, so this weekend I put together a little program that takes BLS data and turns it into csv format, which can be opened in Excel.

    Finally, I would really appreciate two things. First, please mention me (Matthias Shapiro) if you use the data for professional purposes, link back here to let people know where to get it. Second, if this is actually helpful data, please consider tossing $5 or so into my digital hat. I make maybe $50 per year from this blog, so anything to let me know this is worth doing is helpful to me.




    \"\"

    Or you could buy a copy of my book Beautiful Visualization\"\" (disclaimer: I only wrote one chapter, but I call it \”my book\” anyway because that makes me feel important).

    Download employment status (A Tables) (BLS link)

    • 1948 – Nov 2011
    • civilian population
    • labor force
    • participation rate
    • employed
    • employment-to-population ratio
    • unemployed
    • unemployment rate
    • not in labor force
    • persons who currently want a job
    • 1939-Nov 2011
    • payroll job counts for 150 industries/sub-industries
    • csv file (headers labeled \”[state] – [field]\” Example: \”Alabama – Unemployed\”
    • xlsx file (better headers, grouping states)
    • 1976 – Oct 2011
    • labor force by state
    • employed by state
    • unemployed by state
    • unemployment rate by state
    Download state payrolls by industry (BLS Link)

  • 11 Reasons Occupy Wall Street Should Become Occupy Foreclosure

    I was speaking with Brendan Loy the other day and he made the comment (I paraphrase):

    Usually when you perform civil disobedience, the act you\’re performing is in direct relation to the injustice you\’re protesting. For example, Rosa Parks refusing to give up her seat on the bus was in direct defiance of an unjust law requiring her to do exactly that. With the Occupy protesters, they\’re against corporate greed, so they\’re camping in a park. I don\’t get that.

    And I think he\’s right. There seems to be a very loose relationship between what the protesters say they want and their method of protesting.

    Giving this some thought, I think there is an civil disobedience action the Occupiers can take that would make a great deal more sense. And that is occupying foreclosures.

    Hear me out here… I\’m not the most sympathetic toward the Occupy movement, but occupying foreclosures has the following benefits:

    1. Real shelter means fewer deaths (as long as they don\’t do drugs).
    2. The action is directly related to the financial sector (although they would quickly discover that Fannie Mae and Freddie Mac are bigger culprits than Goldman Sachs).
    3. It would be genuinely disruptive to the financial sector. Don\’t fool yourselves, sleeping in a park is more disruptive to a bagel shop than to a hedge fund manager.
    4. Far less impact on small businesses whose owners just want to make ends meet.
    5. They could actually get arrested for peaceful civil disobedience (trespassing) rather than for jaywalking or public indecency.
    6. Good optics if they keep the houses clean & leave when they are sold. Local news pieces would relate directly to real neighborhoods, get great pictures of people and the houses they occupy. People could go check out the movement without heading downtown… the movement is right down the street.
    7. Build excellent community standing (if they are actually good community members in these neighborhoods).
    8. A good platform for spreading their position. If people come to see the houses for purchase, they can pass out literature about the pitfalls of tricksy banks and dangerous mortgages.
    9. They can attach themselves closely to the individual stories of woe within the local community. Every foreclosure comes with a story. They could take advantage of that.
    10. If banks decided it would be better to sell foreclosures for a loss rather than risk an occupation, it might move inventory, actually help solve one of the problems.
    11. Filter out the antagonistic element from Occupy. I suspect anarchists are less interested in playing house with a half dozen people than with running down the streets smashing windows.
    Of course any movement is only as good as the people who are involved with it. But this path seems more targeted, sustainable and sanitary. And it might just be the best place to go next for Occupy.
    Would I support this? Meh. Probably not wholeheartedly. It is still against the law (but civil disobedience is, by definition, against the law). And I\’m sure there are some unintended consequence that I\’ve failed to consider (there always are).
    But at least it would make some kind of sense.

  • How To Cherry Pick Data

    In his post \”Senate Republicans Block Targeted Jobs Relief for Teachers And First Responders\”, Matthew Yglesias points out that \”during the Obama years\” private employment has rebounded while government employment has seen a \”sharp contraction\”.

    Yglesias points to a couple of charts, but I\’ve helpfully replicated his data set into a single chart, because that\’s just the kind of guy I am.

    \"\"

    As you can see, using January 2009 as our point of reference, private jobs have rebounded from a drop of 3.79% in 2010 to a drop of 1.63% in August (my data is slightly out of date, but good enough for gov\’t work… get it?!?). Local gov\’t employment has fallen 3.6% in that same time frame. I also added federal gov\’t employment (which has fallen 2.75% since January 2009) for the heck of it.

    In the comments section, Peter Schaeffer complains that Yglesias is cherry picking the data and points out that gov\’t employment saw +10% gains in the decade leading up to the crash and 3-4% losses from the peak while the private sector saw slightly less than 5% gains in that time period and slightly more than 5% losses from the peak.

    I thought that Schaeffer had a good point, but needed some visuals to drive it home, so I thought I\’d show Yglesias\’ jobs data in Schaeffer\’s context.

    \"\"

    As you can see, Yglesias\’ data starts at a really handy place for his argument, since it begins measuring job losses and growth at a time when we had already seen drastic private sector losses, but no public sector losses.

    Of course, the funny aspect to this data is that one could use it to say that President Obama is reigning in the public sector that George W. Bush let grow out of control. I think the only reason no one is saying this is because everyone on President Obama\’s side would consider that a bad thing and everyone who opposes President Obama would consider that a good thing. Neither side really wants to attribute this trend to President Obama. In fact, President Obama is working actively to reverse this trend.

    Ah, the little ironies of life.

    Note: In the spirit of \”never attribute to malice what can be explained by incompetence\”, I wouldn\’t be surprised if Yglesias unwittingly cherry-picked the data. \”The Obama years\” is a perfectly rational place to start looking at data and, if that was the only data you looked at, it would support his conclusion. On the other hand, Yglesias has always had a better grasp of the data than this particular post suggests, so I suspect he kind-of-sort-of knew that this was a cherry picked sample set but was OK with using it because it bolstered his argument.

  • How To Read Unemployment Reports

    Every time a national unemployment report comes out, I tweet the many details from @politicalmath. Frequently I get a lot of the same questions, so I thought I\’d jot down a quick summary on unemployment reports and numbers and where they come from.

    There are 2 kinds of employment numbers, summarized here:

    1. Establishment Data (Current Employment Statistics or CES) – this survey covers 400,000 businesses and counts the number of payroll positions that are filled.
    2. Household Data (Current Population Survey or CPS) – this survey covers 60,000 households and counts the number of people who are employed and unemployed.

    When an employment report comes out from the Bureau of Labor Statistics (BLS), they usually report:

    1. The unemployment rate, which is calculated using household data
    2. The number of jobs added, which comes from the establishment data

    Sometimes this data can seem contradictory. For example, between March and  June 2011, we gained 290,000 jobs but the unemployment rate went up .4% (from 8.8% to 9.2%).

    There can be a couple reasons for this. The first one is that, the \”jobs added\” number comes from subtracting last month\’s establishment jobs number from this month\’s establishment jobs number, but we never use either of those numbers to calculate the unemployment data.

    Why?

    Because the essence of the establishment jobs number is asking employers: \”How many people work for you?\” It gives a nice accurate number, but it doesn\’t tell us anything about how many people don\’t work for them. We don\’t have any number on the unemployed, only a number for jobs.

    For unemployment, we have to go to individuals and ask them: \”Are you employed or unemployed?\” Then we take the unemployed number and divide it by the total number of people who are in the labor force, which counts both the employed and the unemployed.

    But even the differences between the establishment jobs number and the household jobs number can be big. According to the household jobs number (which is supposed to exclude farm workers and the self-employed), we had 139.6 million jobs in August 2011. According to the establishment jobs number, we had 131.1 million.

    That\’s a difference of 8.5 million jobs, and that kind pf spread is pretty normal. The variation changes a little month-to-month, but we could get a report of  jobs created from the household number and jobs lost from the establishment number. In fact, we saw something similar in August where the household number said we gained 331,000 jobs, but the establishment number said we gained 0.

    So why is the establishment number reported?

    Because the establishment survey is so much larger, more reliable and gives more consistent results. In the graph below , we can see that even though the establishment data counts fewer jobs, it is a less erratic count.

    \"\"

    So… that is a quick explanation of the employment report. I dig into this data once a month, so I\’m pretty familiar and I\’m delighted to answer questions or explain in greater detail in the comments.

  • [FIXED] Three Charts To E-Mail Your Right Wing Brother-In-Law

    Dear goodness, not again.

    I had a nice healthy rant all written for this because people who use charts and data to lie piss me off and the self-righteous ones are the worst. But it detracted from this post, so if you want to, you can read it here. Not work that I\’m proud of, but it\’s fun to write every once in a while.

    There is a piece called \”The Three Charts to E-Mail Your Right Wing Brother-In-Law\” that is making the rounds and impressing many people who don\’t know too much about the underlying data. Which is almost everyone.

    So lets dig into these charts and how we can fix them.

    The first one is about Federal Spending and claims that \”Bush Spending\” saw an 88% increase while Obama spending has seen only a 7.2% increase.

    \"Bush-Obama

    The problems with this chart in no particular order:

    Bush was not responsible for all of 2009 spending

    These two charts assume that the entirety of the 2009 fiscal situation lies squarely on George W. Bush\’s shoulders. I would like to posit that this is unfair. There was a bill that got passed (you may have heard of it) that goes by the popular name \”the stimulus\”. It started immediately spending vast sums of money starting in the fiscal year 2009. George W. Bush had nothing to do with this bill.

    I did a little digging and found that the budget Bush proposed for 2009 was for $3.09 trillion while the amount spent during that fiscal year was $3.52 trillion. Now, this might not matter if these kinds of variations were common. But here is a graph of the difference between the proposed spending and the actual spending for the past 10 years. We\’re going to play a game called \”one of these things is not like the others\”.

    \"\"

    We can see that 2009 is a huge outlier… the difference between what was proposed and what was spent is 5 times more than any other year ( $429.1 billion).

    Yep… that\’s what happens when you propose vast amounts of immediate spending in the middle of a fiscal year. Given that Bush had to sign the budget he was given by a Democratic Congress, I think it\’s charitable to say that he is \”responsible\” for what he proposed: the original $3.09 trillion.

    Data is not adjusted for inflation

    This is a minor quibble, but it matters because it\’s a sign that the person who created the chart doesn\’t care about accuracy. Ignoring inflation will always make spending increases look drastic because we\’re compounding real increases with inflation increases. It also matters because, if we adjust for inflation and use Bush\’s last spending proposal, he increased spending by 39% or about 5% a year.

    The chart stops tracking data at a very convenient place

    President Obama\’s budget proposal basically has us maintaining a stable level of spending until 2014, when it starts increasing drastically. The author chose not to chart this data, even though it was right there in front of him. Why? I assume it\’s because he\’s a partisan hack, but I\’m not altogether prepared to rule out that he is, in fact, just an idiot.

    By including these spending targets, we get a much more \”apples to apples\” comparison where we\’re comparing 8 years of \”Bush spending\” to 7 years of \”Obama spending\”.

    If we take all these problems and put them together, we end up with another chart altogether.

    \"\"

    Chart 2

    The second chart says that Bush increased the deficit and Obama is decreasing it.

    \"Bush-Obama

    First of all, the same \”Bush is responsible for everything in FY2009\” thing above applies here too. In addition to that:

    The stimulus was front-loaded with tax cuts

    I know that right wingers will maintain till their dying breath that tax cuts don\’t reduce revenue, they increase revenue. I\’m not really in that camp and this is my blog, so I get to do things my way. So there.

    According to CNN at the time, the stimulus was going to save the average household $1,179. Using the 2009 Census estimate of 112.6 million households, that comes out to $132.7 billion. If we add that to the $429 billion difference between Bush\’s spending proposal and the spending reality and then subtract that from the final deficit, we get a deficit of $894.4 billion.

    $132.7 billion in stimulus tax cuts
    + $429.1 billion in un-planned spending
    – $1,415.7 billion actual deficit
    ======================
    $836.2 billion of the 2009 deficit that is \”Bush\’s fault\”

    All of the reductions are in the future

    Notice how the chart goes down in 2012 and 2013? Notice how neither of those years have happened? This is because President Obama\’s 2012 budget has made some pretty incredible claims.

    To look at these claims with our feet on the ground, let\’s first look at a revenue chart.

    \"\"

    This is a chart that shows the increase and decrease of federal revenue changes over a 12 month collection period. We can see that recessions mean revenues decrease by as much as 15% year-to-year and that in boom times they can increase by a little over 10% year-to-year. The biggest increase we\’ve ever seen was 12% year-to-year increase (from the 2004 fiscal year to the 2005 fiscal year).

    Now this is the same chart including the revenue increases that the Obama budget proposal assumes will happen.

    \"\"

    Now that is some f***ing audacious hope right there.

    The Obama budget assumes for the sake of future budget planning that we will blow 30 years of revenue data out of the water by clocking in a 21% revenue increase in 2012 and a 14% revenue increase in 2013. Then they assume things will \”calm down\” to a stable 7-8% annual increase, which is merely massive (as opposed to completely insane).

    This is a particularly important point because the estimates that the Obama team made were not just optimistic. They assume we are on some kind of federal revenue breakthrough unheard in this generation.

    The revenue assumptions in this budget proposal have sped right past optimism and into delusion.

    For the sake of fixing this second chart, I am going to be incredibly generous and assume that we see 9% revenue growth over the next 4 years. This would be very good news for our deficit situation and is extremely unlikely. It is not, however, technically impossible, so we\’ll give some benefit of the doubt there.

    Fixing Chart 2

    Accounting for these issues, assuming that we hit the spending targets we\’re aiming for (a big if but one I\’m willing to let it slide) here is the second chart updated.

    \"\"

    Chart 3:

    \"Bush-Obama-Jobs-Chart\"

    Permutations of this chart have been around for some time. President Obama\’s team first started using it in mid 2009 to promote the idea that the stimulus was working. It\’s actually the most honest of the charts here, but there are still some problems with it.

    Using Only Establishment Private Jobs Data

    This makes things look a little better because we\’ve been losing public sector jobs over the last year or two. I\’m not saying \”counting only private sector jobs is an invalid measurement\”. What I am saying is that it is a red flag that the person may be cherry-picking data to get the best result.

    As for using establishment data instead of household survey data, there\’s nothing particularly wrong with that, but it is good to note that the household survey counts about 10 million more jobs and  covers people who are employed but not on a payroll, so it will give a somewhat more complete picture of the employment situation. And, unsurprisingly, the data doesn\’t look quite as good for Obama. It\’s not particularly bad… it\’s just \”meh\”.

    It\’s Bush\’s Fault Only When It\’s Bad

    But the funniest thing about this chart? The author has spent the last 2 charts convincing us that EVERYTHING that happened in the 2009 fiscal year was Bush\’s fault. In this chart, the tune has changed entirely because, if the author gave Bush credit to the end of the 2009 fiscal year, it would look like Bush saved the day. The most drastic reductions in job loss would then fall under the \”Bush\’s fault\” umbrella.

    And we can\’t have that. When it comes to a choice between honest consistency and making George W. Bush look bad, the author didn\’t even blink. So, in a move that is so dishonest is is actually funny, the chart author basically says, \”All jobs saved are due to President Obama and his courageous stimulus, but I blame George W. Bush for all the stimulus spending and stimulus tax cuts that created those jobs.\”

    I created a alternate version of this chart that represents my complaints listed above, but I want to make note that, while I feel the previous \”fixes\” are a better representation of reality, this chart is not nearly as fair as those were. I personally prefer the BLS household data (which I used in this chart) over the payroll data (which the original chart author used), but I\’m not comfortable giving Bush credit for stopping job losses 9 months after he left office. I\’m representing it this way only because I want to give an indication of how the author would have done it if he or she maintained an internal consistency.

    \"\"

  • Alternate Intro to the “Three Charts” Post

    This was my original intro to the “Three Charts” post that I’ve just put up. Normally my rational brain gets the better of me and I delete this kind of crap before I post it. It is neither charitable nor professional. You’ve been warned.

    Once again, the truth rolls its eyes and starts tying it’s laces while the lie is half-way around the world. There is a piece called “The Three Charts to E-Mail Your Right Wing Brother-In-Law” that is making the rounds and impressing many people who didn’t score high on the “critical thinking” portion of the test.

    Whoever made this chart is an asshole. Straight up asshole. “Your Right Wing Brother-In-Law”? Really? I mean… I get it because no one who is related to this person by blood could ever be so stupid. Nice little undercurrent enforcing the concept of hereditary intelligence. Yeah… I heard a lot of that kind of shit when I visited the deep South and ran into the local racist assholes. Just enough plausible deniability to claim innocence, but deep down you know that, due to your pedigree, you’re just better than other people.

    And someone who disagrees with you could never be your friend. I mean… you? Voluntarily associate with someone who doesn’t agree with you? Actually have friends who might challenge you on something? Heaven forbid. Disagreeing with you is something only the lowly and inferior do and you would never be around them of your own volition, so it has to be someone who is “in the family, but not part of the family”.

    No, the right wing brother in law just mouths off without knowing the facts, so it is up to the great brainiac hope of the family to inject reality into the conversation. Is that a banana in your pocket or are you just fantasizing about winning an argument in real life without stuttering and turning red in the face?

    And as salty icing on the moldy self-righteous cake why do YOU need to explain this to the “brother-in-law” specifically? It it because your wife or sister is too stupid or too meek to explain it to him herself? The weak-minded damsel will be saved by you, the powerful, witty, urbane, strong, righteous intellectual knight. And everyone will love you for being the big savior of the family conversation.

    I find that creepy, dismissive and sexist. Everyone who giggles at it and re-shares it should be ashamed of themselves but won’t be because they aren’t introspective enough to stop stroking their own intellectual egos for the time it takes to feel shame.

    See? This is why I usually delete that kind of stuff.

  • I’m Still Catching Up

    I want to send out this apology. Policy analysis is not my job… I just do this for fun. That last post was meant to explain things I couldn’t explain over Twitter (to an audience of about 15 people who already discussing it).

    Instead, I got almost as many comments and hits as I’ve gotten on everything else I’ve done here over the past 2 years. I’m trying to digest it all, but it’s crunch time at my “normal job”.

    Thank you to everyone who has commented and engaged the topic. I am going to try to do a follow-up in the coming week that addresses more of the data.