Abstract
The NYC Department of Health and Mental Hygiene (DOHMH) makes several methodological claims regarding our paper that we would like to address, point-by-point, in this response. We then address their broader points on collaboration, open data, and their commitment to anti-racist practices.
Overall, we believe that none of their points regarding our statistical analyses influence the overall finding of our study: there is evidence of inequalities in inspectors’ ratings only for Asian-cuisine restaurants after the onset of news of the COVID-19 pandemic: (1) The first methodological claim is that they searched for evidence of bias against Asian restaurants in their inspectional data and were unable to replicate the findings. The DOHMH shows a table of descriptive statistics of scores and violations by cuisine. First, we are clear in our paper that we used the average of scores since 2017, by cuisine. Our rationale of using the average of the period instead of the average of the month is to smooth the distributions to examine whether there was a considerable uptick between December 2019 and January 2020. Second, after smoothing the distributions, we apply the synthetic control method to predict what would have been the average scores post December for every cuisine. To our knowledge the DOHMH did none of the two main things to replicate the analysis, analyze the cumulative average scores and apply a causal strategy to identify an effect. Stating they tried to replicate our analysis without finding the same results is not just misleading, but false. (2) The next claims are concerned with the use of the data, which according to the DOHMH is invalid for historical analysis. The reasons they cite are two: (a) Their first claim is that older data over-represents more established restaurants and newer data over-represents less established ones, as more established restaurants tend to do better on average in terms of ratings. We want to note here that this is true for all cuisines as the trend of cumulative scores show in our paper, but if newer data ‘causes’ worse performance due to new non-established restaurants, that is the case for all cuisines, not just for Asian restaurants. (b) Their second claim is that patterns of openings and closings are not at random and show an example for Asian and American restaurants: in their figure, more Asian restaurants opened recently than American ones. This may be–and likely is–true for other cuisines as well, but DOHMH did not show the percentage of “newer” restaurants for the other cuisines. More importantly, the synthetic control method controls for these so-called unobserved variables, since “Only units that are alike in both observed and unobserved determinants of the outcome variable as well as in the effect of those determinants on the outcome variable should produce similar trajectories of the outcome variable over extended periods of time” (Abadie et al., 2015).
1
(3) Related to point 2, we believe when the DOHMH refers to historical analysis they assume we are investigating trends of specific restaurants which open and close all the time in NYC. We are not. In our study, we are investigating trends of cuisines, and therefore use the cumulative averages through time by cuisine, and not by restaurant, to understand trend behavior.
2
Stating that one cannot analyze trends of scores by cuisine is analogous to stating one cannot analyze GDP by country through time because some companies are not in business anymore. Both are an aggregate measure, and our intention was to make an aggregate analysis through time, not a restaurant-specific one. (4) DOHMH claims we used a view of the data made available by an anonymous user, which we did not. The dataset to which they refer has a disclaimer that “a member of the public created this dataset. NYC Open Data has not reviewed or endorsed any changes…” This dataset has “93K” rows. Our dataset has 200,539 rows; therefore, their claim is incorrect. (5) The DOHMH mentions we did not use as a control in our analysis whether restaurants are chains, which mostly correlated with American restaurants. In this case, this control would similarly affect Caribbean, Mexican or Latin restaurants but when we perform placebo tests for all cuisines, we did not observe an effect on any of them. (6) Related to the previous point and the point regarding lack of control by newness of the restaurant type, we find it troublesome that the DOHMH centers all of their claims on American versus Asian restaurants. They note American restaurants perform better, but in our analyses, we include cuisines that do not perform better, on average, than Asian restaurants. The point of the synthetic control method is to find a combination of weights of all the groups used that properly predict what would happen before and after a shock. A similarity in the trends of the outcome variable is not going to be observed unless we include cuisines that are similar to Asian, as mentioned in point 2b. Moreover, we are not only comparing Asian, versus American cuisines, but to Caribbean, Mexican, Italian and Latin as well, and we find that even among cuisines that perform worse than Asian restaurants do, no effect is observed.
In the following, we address other points raised by the DOHMH. • The DOHMH states that we did not collaborate with them in our analyses, which is true. It is common practice that university researchers, such as ourselves, use open data in ways that complement the day-to-day analyses of government agencies. To work together on analyses of data could present a severe conflict of interest, as academic research is beholden to standards of objectivity (to the extent possible). An exception could be if the open data was somehow unclear, which would influence subsequent analyses. However, through multiple communications with the DOHMH and the information provided in their rejoinder, we have learned nothing that would prevent us from doing our analyses again. • Relatedly, the DOHMH states multiple times that “such heterogeneous data” is “an unpredictable mix of pre- and post-adjudicated violations at any given time.” While we understand that data is often ‘living’ and refreshed, to strike a tone that the data should not be used in non-superficial ways seems to contradict the spirit of New York City’s Open Data Law: “Local Law 11 of 2012 the ‘Open Data Law,’ mandates that city agencies make their data available to the public via the Open Data Portal.”
3
• It is good to hear that the New York City Department of Health and Mental Hygiene is publicly committed to anti-racist practices, as indicated by the website they provide in their response. Using websites that track publication dates of websites, we found that the first published incidence of this website was on May 12, 2020, which was after the first draft of the study was finished. More substantive information (such as the “Race to Justice Action Kit”) beyond a statement similar to the one provided in the rejoinder and an email address was not added until April 4, 2021. It is notable that nowhere in any of the current or historic material is there any direct mention of restaurant health inspections.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Data availability statement
Please follow this link (Abadie et al., 2015) for a version of the 
used in this article/rejoinder.