The chi squared test is commonly used to test the homogeneity of two binomial proportions. We will show by accurate computations and analytical proof that even for very large sample sizes the exact size of this test can be nearly 80% larger than the nominal level. For example, a nominal .05 test can have a size of over .09. Whether the test has this inflated size depends only on the ratio of the binomial sample sizes. A closer examination of the error properties of this test may indicate that this inflation of size may not be of great concern. We compare the size of the chi squared test to an exact unconditional test that has remarkably good size properties.