Stata version bug
Opened this issue · 14 comments
johnklopfer commented
Hi, I installed your code exactly as described on this page. When I enter the following command line in Stata, I get an error, but the code runs fine in R, and the package the error message refers to is installed. In general, the Stata wrapper seems to have install errors for package dependencies in R.
I'll just run the code in R for now, but thought you might want to know about this!
. staggered, y("complaints") g("first_trained") t("period") i("uid") estimand("simple")
Installing devtools and staggered packages in R
Error in install.packages("devtools", repos = "http://cran.us.r-project.org") : unable to install packages
jonathandroth commented
Hi, thank you for bringing this to our attention!
Can you try running
install.packages("devtools", repos = "http://cran.us.r-project.org")
in the R terminal and let me know if it works? I am trying to figure out
whether this is an issue on the R end or Stata end.
…On Thu, Aug 5, 2021 at 8:50 AM johnklopfer ***@***.***> wrote:
Hi, I installed your code exactly as described on this page. When I enter
the following command line in Stata, I get an error, but the code runs fine
in R, and the package the error message refers to is installed. In general,
the Stata wrapper seems to have install errors for package dependencies in
R.
I'll just run the code in R for now, but thought you might want to know
about this!
. staggered, y("complaints") g("first_trained") t("period") i("uid")
estimand("simple")
Installing devtools and staggered packages in R
Error in install.packages("devtools", repos = "
http://cran.us.r-project.org") : unable to install packages
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#10>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
.
johnklopfer commented
For sure - thank you for writing software!
That command line works, and the bug in Stata (same error message, still trying to install devtools and staggered) is still there.
John
… On Aug 6, 2021, at 3:42 AM, Jonathan Roth ***@***.***> wrote:
Hi, thank you for bringing this to our attention!
Can you try running
install.packages("devtools", repos = "http://cran.us.r-project.org")
in the R terminal and let me know if it works? I am trying to figure out
whether this is an issue on the R end or Stata end.
On Thu, Aug 5, 2021 at 8:50 AM johnklopfer ***@***.***> wrote:
> Hi, I installed your code exactly as described on this page. When I enter
> the following command line in Stata, I get an error, but the code runs fine
> in R, and the package the error message refers to is installed. In general,
> the Stata wrapper seems to have install errors for package dependencies in
> R.
>
> I'll just run the code in R for now, but thought you might want to know
> about this!
>
> . staggered, y("complaints") g("first_trained") t("period") i("uid")
> estimand("simple")
> Installing devtools and staggered packages in R
> Error in install.packages("devtools", repos = "
> http://cran.us.r-project.org") : unable to install packages
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#10>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
> .
>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQMZRGPYJCMGX6BG7C4ZYM3T3LSSDANCNFSM5BTZNY3A>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.
johnklopfer commented
I tinkered around a little more today and can add -
1. The problem was with RCall to command line R. After I installed RStudio and all of the dependencies, the Stata version works without any error messages.
2. staggered in RStudio is throwing the warning message:
Warning message:
In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", :
The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007, 2008, 2010, 2011 have a single cross-sectional unit only. We drop these cohorts.
after I used command staggered(df = short,
+ i = "FIPS",
+ t = "SURVYEAR",
+ g = "YSFR1",
+ y = "PMTE11",
+ estimand = "simple”)
on the attached file:
3. staggered in Stata produces the same result, but does not throw the warning message.
4. staggered_cs in Stata (estimand simple) throws the same warning message and produces
'estimate' 'se' 'se_neyman'
'simple' 178.26605 343.76784 409.91009
while csdid in Stata (estimand simple) produces
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
ATT | 423.6928 384.5374 1.10 0.271 -329.9867 1177.372
5. If I drop the cohorts named in the warning message, csdid in Stata (estimand simple) produces
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
ATT | 161.1675 386.1504 0.42 0.676 -595.6734 918.0084
I think the major difference here is the dropped cohorts, though I know there are other differences between the two programs. Is there a reason to drop cohorts with one cross-sectional unit, when 2x2s are going to be aggregated later? It would be great to be able to use the full dataset without disaggregating from the FIPSxYEAR level I’m using here!
Hope some of this is useful :)
John
… On Aug 6, 2021, at 8:39 AM, John Klopfer ***@***.***> wrote:
For sure - thank you for writing software!
That command line works, and the bug in Stata (same error message, still trying to install devtools and staggered) is still there.
John
> On Aug 6, 2021, at 3:42 AM, Jonathan Roth ***@***.*** ***@***.***>> wrote:
>
>
> Hi, thank you for bringing this to our attention!
>
> Can you try running
>
> install.packages("devtools", repos = "http://cran.us.r-project.org <http://cran.us.r-project.org/>")
>
> in the R terminal and let me know if it works? I am trying to figure out
> whether this is an issue on the R end or Stata end.
>
> On Thu, Aug 5, 2021 at 8:50 AM johnklopfer ***@***.***> wrote:
>
> > Hi, I installed your code exactly as described on this page. When I enter
> > the following command line in Stata, I get an error, but the code runs fine
> > in R, and the package the error message refers to is installed. In general,
> > the Stata wrapper seems to have install errors for package dependencies in
> > R.
> >
> > I'll just run the code in R for now, but thought you might want to know
> > about this!
> >
> > . staggered, y("complaints") g("first_trained") t("period") i("uid")
> > estimand("simple")
> > Installing devtools and staggered packages in R
> > Error in install.packages("devtools", repos = "
> > http://cran.us.r-project.org <http://cran.us.r-project.org/>") : unable to install packages
> >
> > —
> > You are receiving this because you are subscribed to this thread.
> > Reply to this email directly, view it on GitHub
> > <#10 <#10>>, or unsubscribe
> > <https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A <https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A>>
> > .
> > Triage notifications on the go with GitHub Mobile for iOS
> > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>>
> > or Android
> > <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>>
> > .
> >
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQMZRGPYJCMGX6BG7C4ZYM3T3LSSDANCNFSM5BTZNY3A>.
> Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.
>
pedrohcgs commented
Hi John,
Would you be able to share the data (or subset) so we can dig into this
more closely?
You can send to my email, if you want to.
Thanks
On Fri, Aug 6, 2021 at 04:14 johnklopfer ***@***.***> wrote:
I tinkered around a little more today and can add -
1. The problem was with RCall to command line R. After I installed RStudio
and all of the dependencies, the Stata version works without any error
messages.
2. staggered in RStudio is throwing the warning message:
Warning message:
In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", :
The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007,
2008, 2010, 2011 have a single cross-sectional unit only. We drop these
cohorts.
after I used command staggered(df = short,
+ i = "FIPS",
+ t = "SURVYEAR",
+ g = "YSFR1",
+ y = "PMTE11",
+ estimand = "simple”)
on the attached file:
3. staggered in Stata produces the same result, but does not throw the
warning message.
4. staggered_cs in Stata (estimand simple) throws the same warning message
and produces
'estimate' 'se' 'se_neyman'
'simple' 178.26605 343.76784 409.91009
while csdid in Stata (estimand simple) produces
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
ATT | 423.6928 384.5374 1.10 0.271 -329.9867 1177.372
5. If I drop the cohorts named in the warning message, csdid in Stata
(estimand simple) produces
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
ATT | 161.1675 386.1504 0.42 0.676 -595.6734 918.0084
I think the major difference here is the dropped cohorts, though I know
there are other differences between the two programs. Is there a reason to
drop cohorts with one cross-sectional unit, when 2x2s are going to be
aggregated later? It would be great to be able to use the full dataset
without disaggregating from the FIPSxYEAR level I’m using here!
Hope some of this is useful :)
John
> On Aug 6, 2021, at 8:39 AM, John Klopfer ***@***.***> wrote:
>
> For sure - thank you for writing software!
> That command line works, and the bug in Stata (same error message, still
trying to install devtools and staggered) is still there.
> John
>
>> On Aug 6, 2021, at 3:42 AM, Jonathan Roth ***@***.*** ***@***.***>>
wrote:
>>
>>
>> Hi, thank you for bringing this to our attention!
>>
>> Can you try running
>>
>> install.packages("devtools", repos = "http://cran.us.r-project.org <
http://cran.us.r-project.org/>")
>>
>> in the R terminal and let me know if it works? I am trying to figure out
>> whether this is an issue on the R end or Stata end.
>>
>> On Thu, Aug 5, 2021 at 8:50 AM johnklopfer ***@***.***> wrote:
>>
>> > Hi, I installed your code exactly as described on this page. When I
enter
>> > the following command line in Stata, I get an error, but the code
runs fine
>> > in R, and the package the error message refers to is installed. In
general,
>> > the Stata wrapper seems to have install errors for package
dependencies in
>> > R.
>> >
>> > I'll just run the code in R for now, but thought you might want to
know
>> > about this!
>> >
>> > . staggered, y("complaints") g("first_trained") t("period") i("uid")
>> > estimand("simple")
>> > Installing devtools and staggered packages in R
>> > Error in install.packages("devtools", repos = "
>> > http://cran.us.r-project.org <http://cran.us.r-project.org/>") :
unable to install packages
>> >
>> > —
>> > You are receiving this because you are subscribed to this thread.
>> > Reply to this email directly, view it on GitHub
>> > <#10 <
#10>>, or unsubscribe
>> > <
https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A
<
https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A
>>
>> > .
>> > Triage notifications on the go with GitHub Mobile for iOS
>> > <
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
<
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
>>
>> > or Android
>> > <
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
<
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
>>
>> > .
>> >
>> —
>> You are receiving this because you authored the thread.
>> Reply to this email directly, view it on GitHub <
#10 (comment)>,
or unsubscribe <
https://github.com/notifications/unsubscribe-auth/AQMZRGPYJCMGX6BG7C4ZYM3T3LSSDANCNFSM5BTZNY3A
>.
>> Triage notifications on the go with GitHub Mobile for iOS <
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android <
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
>.
>>
>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABE734YTWH2R2A25LHEPUM3T3O7Z7ANCNFSM5BTZNY3A>
.
--
===============================
Pedro H. C. Sant'Anna
Department of Economics
Vanderbilt University
615-875-8448 (phone)
***@***.***
https://pedrohcgs.github.io
===============================
johnklopfer commented
Hmm I guess Github drops attachments - here it is again!
Thank you both,
John
… On Aug 6, 2021, at 10:34 PM, pedrohcgs ***@***.***> wrote:
Hi John,
Would you be able to share the data (or subset) so we can dig into this
more closely?
You can send to my email, if you want to.
Thanks
On Fri, Aug 6, 2021 at 04:14 johnklopfer ***@***.***> wrote:
> I tinkered around a little more today and can add -
>
> 1. The problem was with RCall to command line R. After I installed RStudio
> and all of the dependencies, the Stata version works without any error
> messages.
>
> 2. staggered in RStudio is throwing the warning message:
>
> Warning message:
> In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", :
> The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007,
> 2008, 2010, 2011 have a single cross-sectional unit only. We drop these
> cohorts.
>
> after I used command staggered(df = short,
> + i = "FIPS",
> + t = "SURVYEAR",
> + g = "YSFR1",
> + y = "PMTE11",
> + estimand = "simple”)
>
> on the attached file:
>
>
>
> 3. staggered in Stata produces the same result, but does not throw the
> warning message.
>
> 4. staggered_cs in Stata (estimand simple) throws the same warning message
> and produces
>
> 'estimate' 'se' 'se_neyman'
> 'simple' 178.26605 343.76784 409.91009
>
> while csdid in Stata (estimand simple) produces
>
> | Coef. Std. Err. z P>|z| [95% Conf. Interval]
>
> -------------+----------------------------------------------------------------
> ATT | 423.6928 384.5374 1.10 0.271 -329.9867 1177.372
>
> 5. If I drop the cohorts named in the warning message, csdid in Stata
> (estimand simple) produces
>
> | Coef. Std. Err. z P>|z| [95% Conf. Interval]
>
> -------------+----------------------------------------------------------------
> ATT | 161.1675 386.1504 0.42 0.676 -595.6734 918.0084
>
> I think the major difference here is the dropped cohorts, though I know
> there are other differences between the two programs. Is there a reason to
> drop cohorts with one cross-sectional unit, when 2x2s are going to be
> aggregated later? It would be great to be able to use the full dataset
> without disaggregating from the FIPSxYEAR level I’m using here!
>
> Hope some of this is useful :)
>
> John
>
>
> > On Aug 6, 2021, at 8:39 AM, John Klopfer ***@***.***> wrote:
> >
> > For sure - thank you for writing software!
> > That command line works, and the bug in Stata (same error message, still
> trying to install devtools and staggered) is still there.
> > John
> >
> >> On Aug 6, 2021, at 3:42 AM, Jonathan Roth ***@***.*** ***@***.***>>
> wrote:
> >>
> >>
> >> Hi, thank you for bringing this to our attention!
> >>
> >> Can you try running
> >>
> >> install.packages("devtools", repos = "http://cran.us.r-project.org <
> http://cran.us.r-project.org/>")
> >>
> >> in the R terminal and let me know if it works? I am trying to figure out
> >> whether this is an issue on the R end or Stata end.
> >>
> >> On Thu, Aug 5, 2021 at 8:50 AM johnklopfer ***@***.***> wrote:
> >>
> >> > Hi, I installed your code exactly as described on this page. When I
> enter
> >> > the following command line in Stata, I get an error, but the code
> runs fine
> >> > in R, and the package the error message refers to is installed. In
> general,
> >> > the Stata wrapper seems to have install errors for package
> dependencies in
> >> > R.
> >> >
> >> > I'll just run the code in R for now, but thought you might want to
> know
> >> > about this!
> >> >
> >> > . staggered, y("complaints") g("first_trained") t("period") i("uid")
> >> > estimand("simple")
> >> > Installing devtools and staggered packages in R
> >> > Error in install.packages("devtools", repos = "
> >> > http://cran.us.r-project.org <http://cran.us.r-project.org/>") :
> unable to install packages
> >> >
> >> > —
> >> > You are receiving this because you are subscribed to this thread.
> >> > Reply to this email directly, view it on GitHub
> >> > <#10 <
> #10>>, or unsubscribe
> >> > <
> https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A
> <
> https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A
> >>
> >> > .
> >> > Triage notifications on the go with GitHub Mobile for iOS
> >> > <
> https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
> <
> https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
> >>
> >> > or Android
> >> > <
> https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
> <
> https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
> >>
> >> > .
> >> >
> >> —
> >> You are receiving this because you authored the thread.
> >> Reply to this email directly, view it on GitHub <
> #10 (comment)>,
> or unsubscribe <
> https://github.com/notifications/unsubscribe-auth/AQMZRGPYJCMGX6BG7C4ZYM3T3LSSDANCNFSM5BTZNY3A
> >.
> >> Triage notifications on the go with GitHub Mobile for iOS <
> https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android <
> https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
> >.
> >>
> >
>
> —
>
>
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#10 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABE734YTWH2R2A25LHEPUM3T3O7Z7ANCNFSM5BTZNY3A>
> .
>
--
===============================
Pedro H. C. Sant'Anna
Department of Economics
Vanderbilt University
615-875-8448 (phone)
***@***.***
https://pedrohcgs.github.io
===============================
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQMZRGIQD7IX45K2B2Y2Z2LT3PXFVANCNFSM5BTZNY3A>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.
pedrohcgs commented
@johnklopfer can you send it to my email? pedro.h.santanna@vanderbilt.edu
Github is blocking it.
pedrohcgs commented
@johnklopfer got it!!
pedrohcgs commented
Some thoughts on some errors:
-
It seems that
FIPSis not a cross-sectional identifier, as we observe several observations of the sameFIPSfor the same time period. Should seti = leaid, asleaidseems to be the correct cross-sectional identifier here. I did this in R and had no problems. -
I just double checked in R that
didandstaggered_csagreed on point estimates. They do. Indid, you need to make sure that you setcontrol_group = "notyettreated". -
I am not getting any error with
staggeredorstaggered_csin R. Would you mind updating/reinstall the R package and see if you still get the error? -
I am also getting the error message on Stata : "vcv_neyman not found'. Need to dig more about why this is happening.
johnklopfer commented
Thanks Pedro!
Quick checks and comments:
0. I think I confused things by sending two datasets: leaid is nested in FIPS, and ’shortlea.csv’ is a disaggregated version of ’short.csv’.
1a. Like you, I had no trouble running the leaid-level regression in R (or in Stata) for the dataset ’shortlea.csv’ (observations at the leaid x year level) that I shared with you.
1b. The trouble is with running the FIPS-level regression in R (or in Stata) for the dataset ’short.csv’ (observations at the FIPS x year level). It’s not clear why R is throwing the warning message:
Warning message:
In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", :
The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007, 2008, 2010, 2011 have a single cross-sectional unit only. We drop these cohorts.
This creates a dilemma - can’t get the right coefficient estimate in the FIPS-level dataset because half the treatment cohorts are dropped, and can’t get the right standard error estimate in the leaid-level dataset because treatment is assigned (and errors should be clustered) at the FIPS level (which can only be accomplished with this package by working at or aggregating up to the FIPS level; per p. 21, Remark 5).
2. Similarly, like you I find that (in Stata) 'csdid, notyet' and ’staggered_cs’ produce identical results with ’shortlea.csv’. The trouble is these same two (nominally equivalent) commands produce different results on ’short.csv’, because of the treatment cohorts csdid does not drop and staggered_cs does drop (see 1b above for the list):
csdid, notyet: coefficient: 450.6242 standard error: 376.2183
staggered_cs: coefficient: 178.26605 standard error: 343.76784
csdid, notyet; dropping same treatment cohorts as staggered_cs: coefficient: 178.2661 standard error: 385.7158
3. I reinstalled ‘staggered' in R. I still get the “We drop these cohorts." error with the file ’short.csv’. Because there are multiple leaid per FIPS in each year, I still do not get the “We drop these cohorts.” error with ’shortlea.csv’.
4. Sure, please let me know what you digging turns up!
John
… On Aug 17, 2021, at 5:39 AM, pedrohcgs ***@***.***> wrote:
Some thoughts on some errors:
It seems that FIPS is not a cross-sectional identifier, as we observe several observations of the same FIPS for the same time period. Should set i = leaid , as leaid seems to be the correct cross-sectional identifier here. I did this in R and had no problems.
I just double checked in R that did and staggered_cs agreed on point estimates. They do. In did, you need to make sure that you set control_group = "notyettreated" .
I am not getting any error with staggered or staggered_cs in R. Would you mind updating/reinstall the R package and see if you still get the error?
I am also getting the error message on Stata : "vcv_neyman not found'. Need to dig more about why this is happening.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQMZRGJEL7CITVLJK5P2S5LT5GAQTANCNFSM5BTZNY3A>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.
pedrohcgs commented
Can you send me the short.csv file?
Thanks!
On Mon, Aug 16, 2021 at 21:37 johnklopfer ***@***.***> wrote:
Thanks Pedro!
Quick checks and comments:
0. I think I confused things by sending two datasets: leaid is nested in
FIPS, and ’shortlea.csv’ is a disaggregated version of ’short.csv’.
1a. Like you, I had no trouble running the leaid-level regression in R (or
in Stata) for the dataset ’shortlea.csv’ (observations at the leaid x year
level) that I shared with you.
1b. The trouble is with running the FIPS-level regression in R (or in
Stata) for the dataset ’short.csv’ (observations at the FIPS x year level).
It’s not clear why R is throwing the warning message:
Warning message:
In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", :
The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007,
2008, 2010, 2011 have a single cross-sectional unit only. We drop these
cohorts.
This creates a dilemma - can’t get the right coefficient estimate in the
FIPS-level dataset because half the treatment cohorts are dropped, and
can’t get the right standard error estimate in the leaid-level dataset
because treatment is assigned (and errors should be clustered) at the FIPS
level (which can only be accomplished with this package by working at or
aggregating up to the FIPS level; per p. 21, Remark 5).
2. Similarly, like you I find that (in Stata) 'csdid, notyet' and
’staggered_cs’ produce identical results with ’shortlea.csv’. The trouble
is these same two (nominally equivalent) commands produce different results
on ’short.csv’, because of the treatment cohorts csdid does not drop and
staggered_cs does drop (see 1b above for the list):
csdid, notyet: coefficient: 450.6242 standard error: 376.2183
staggered_cs: coefficient: 178.26605 standard error: 343.76784
csdid, notyet; dropping same treatment cohorts as staggered_cs:
coefficient: 178.2661 standard error: 385.7158
3. I reinstalled ‘staggered' in R. I still get the “We drop these
cohorts." error with the file ’short.csv’. Because there are multiple leaid
per FIPS in each year, I still do not get the “We drop these cohorts.”
error with ’shortlea.csv’.
4. Sure, please let me know what you digging turns up!
John
> On Aug 17, 2021, at 5:39 AM, pedrohcgs ***@***.***> wrote:
>
>
> Some thoughts on some errors:
>
> It seems that FIPS is not a cross-sectional identifier, as we observe
several observations of the same FIPS for the same time period. Should set
i = leaid , as leaid seems to be the correct cross-sectional identifier
here. I did this in R and had no problems.
>
> I just double checked in R that did and staggered_cs agreed on point
estimates. They do. In did, you need to make sure that you set
control_group = "notyettreated" .
>
> I am not getting any error with staggered or staggered_cs in R. Would
you mind updating/reinstall the R package and see if you still get the
error?
>
> I am also getting the error message on Stata : "vcv_neyman not found'.
Need to dig more about why this is happening.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <
#10 (comment)>,
or unsubscribe <
https://github.com/notifications/unsubscribe-auth/AQMZRGJEL7CITVLJK5P2S5LT5GAQTANCNFSM5BTZNY3A
>.
> Triage notifications on the go with GitHub Mobile for iOS <
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android <
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
>.
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABE73474G4XWTZULVNN5QL3T5HRRNANCNFSM5BTZNY3A>
.
--
===============================
Pedro H. C. Sant'Anna
Department of Economics
Vanderbilt University
615-875-8448 (phone)
***@***.***
https://pedrohcgs.github.io
===============================
pedrohcgs commented
On the Stata bug: We are currently going deep into this because this is a weird bug. Depending on the event time structure, it currently may or may not work. Which doesn't happen in R!
Now, with a single treated cluster: The theory of our paper and also of Callaway and Sant'Anna (and I'd say all papers doing staggered DiD) do not cover that case. Issue is that we can't identify sigma with a single observation; not much we can do on this...
I will keep you posted on the Stata bug. Keeping this open for that.
Thanks
johnklopfer commented
Hi Pedro, not sure I understand on single treated clusters: your Stata implementation of Callaway and Sant’Anna (csdid, agg(group)) plays just fine with singleton treated clusters - drops none, gives standard errors for each! I plan to check a few of the others, but also had no issues with Borusyak, Jaravel, and Spiess (did_imputation) or with Gardner (did2s).
Help me understand the problem? The one place that looks like trouble is the degrees of freedom correction for \hat{S_g} in section 2.5.1. of your paper, which is very different from that for S_g in section 2.4. Is that correct (shouldn’t Equation 5 be estimable for a single treated cluster, for example)? Is there no alternative using a different plug-in estimator, a different approximation to the optimum, etc.?
… On Aug 24, 2021, at 1:30 AM, pedrohcgs ***@***.***> wrote:
On the Stata bug: We are currently going deep into this because this is a weird bug. Depending on the event time structure, it currently may or may not work. Which doesn't happen in R!
Now, with a single treated cluster: The theory of our paper and also of Callaway and Sant'Anna (and I'd say all papers doing staggered DiD) do not cover that case. Issue is that we can't identify sigma with a single observation; not much we can do on this...
I will keep you posted on the Stata bug. Keeping this open for that.
Thanks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQMZRGPK7HJOOKLZ2LPLELLT6KAVFANCNFSM5BTZNY3A>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.
pedrohcgs commented
The theory behind CS relies on the number of units in each group growing. I believe this is the same for BJS and Gardner. The fact that csdid, did_imputation and did2s produce results do not imply that their results are necessarily reliable for this particular case with one unit per group.
As an author of CS, I can talk more about that one. For instance, with a single treated unit in a group, CS essentially only accounts for the uncertainty coming from the construction of the comparison group. That is, it treat Y(g)|G=g as fixed (zero variance).
With staggered, this is trickier bc we are adopting a design-based perspective, and we cant estimate the variance of Y(g) if group G=g has a single observation. Treating the variance as zero is not really a "solution"....
Does this help?
Thanks
jonathandroth commented
Just to follow up on what Pedro said, the challenge is if N_g = 1 (or you
only have 1 cluster for g), there is no way to estimate S_g = Var[ Y(g) ].
The sample variance will be 0 regardless of what the true variance is.
Pedro's csdid package essentially assumes that S_g =0 if N_g = 1. It is not
accounting for the sampling variation from that cohort. I do not think this
is very desirable, because it makes it seem as if the estimates are more
precise than they are likely to be, because it is not accounting for the
uncertainty for cohort g. I am not sure what the other packages are doing,
but I suspect that they are doing something similar. Just because a package
outputs a standard error does not mean that that standard error is
reliable. So I think it is better to throw a warning that we can't estimate
S_g rather than to assume it is zero and give a potentially unreliable
standard error.
If you were willing to impose that S_g = S_{g'} for some g' with more
observations, then you could still form an estimate of the covariance
matrix while included cohort g. If there is demand for others for this
feature, we will consider implementing it. Likewise, randomization
inference will be valid under the sharp null for any test statistic
(including the "studentized" one that imposes S_g =0, but there are no
guarantees for validity for the weak null in the case where N_g = 1).
…On Fri, Sep 10, 2021 at 3:10 PM pedrohcgs ***@***.***> wrote:
The theory behind CS relies on the number of units in each group growing.
I believe this is the same for BJS and Gardner. The fact that csdid,
did_imputation and did2s produce results do not imply that their results
are necessarily reliable for this particular case with one unit per group.
As an author of CS, I can talk more about that one. For instance, with a
single treated unit in a group, CS essentially only accounts for the
uncertainty coming from the construction of the comparison group. That is,
it treat Y(g)|G=g as fixed (zero variance).
With staggered, this is trickier bc we are adopting a design-based
perspective, and we cant estimate the variance of Y(g) if group G=g has a
single observation. Treating the variance as zero is not really a
"solution"....
Does this help?
Thanks
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE6EXFA4OS5DTTSAEYXJPVDUBJJ2NANCNFSM5BTZNY3A>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.