Reproduction code for INCARCERATION, EMPLOYMENT, AND EARNINGS: DYNAMICS AND DIFFERENCES by Grey Gordon, John B. Jones, Urvi Neelakantan, and Kartik Athreya RED-D-21-00319 Loukas Karabarbounis was the coordinating editor OVERVIEW There are 3 main sections that must be reproduced in this order 1. Data See Data/readme.txt (or below) for reproducing those results 2. reproductionMLE See reproductionMLE/ReadMe_MLE.txt (or below) for reproducing those results 3. Fortran and Matlab simulation results Put reproductionFortranMatlab.tar.gz in the folder you want to store lots of results and then extract (using "tar xvzf reproductionFortranMatlab.tar.gz" in the terminal). Then see reproductionFortranMatlab/readme.txt (or below). RUNTIME NOTE Full reproduction in reasonable time requires a cluster. On a single machine, it might take 4-5 months. On a cluster with 100 nodes, all the jobs will take around 2 days. The time scales linearly in nodes. REQUIREMENTS FOR DATA (Step 1 above) Windows Stata 17 (or compatible) REQUIREMENTS FOR REPRODUCTIONMLE (Step 2 above) Windows Matlab R2020b (or compatible) Matlab MEX configured with Visual C++ 2019 (C) REQUIREMENTS FOR FORTRAN AND MATLAB RESULTS (Step 3 above) Linux or Mac Slurm-based cluster (for reasonable run times) Matlab 2021a (or compatible) Intel Fortran 19.0.3 or compatible with an MPI wrapper (mpif90) and associated MPI libraries. MKL libraries, specifically BLAS and LAPACK and VSL are used. 64gb of memory (per node) README FOR DATA CPS The downloaded dataset is cps_00018.dta. First run the files in the "incearn_decile_cutoffs_qreg_nounemp" folder in the following order to obtain earnings decile cutoffs over the life cycle separately for each race X gender x education group: 0.create_race_sex_ed_data.do (69 seconds) 1.calculate_decile_cutoffs.do (3.5 hours) 2.graphs.do (22 seconds) 3.all_cutoffs_merged.do (1 second) Then run the files in the "within_decile_mean_earnings_nounemp" folder in the following order to obtain within-decline mean earnings over the life cycle for each group: 1.within_decile_means_real_by_age.do (6 seconds) 2.loop_over_all_files.do - uses means_smoothing.do (70 seconds) 3.final_means_and_graphs.do (26 seconds) 4.final_data (2 seconds) 5.export_to_excel (1 second) To calculate percentile cutoffs run 1.pctile_cutoffs_basis_points.do (42 minutes) NSLY The downloaded dataset is nlsy_rev3. To obtain the statistics in nlsy_edattain_1979.csv and nlsy_gender_and_race.csv, run nlsy_rev3_desc_stats.do (2 seconds). To obtain the age earnings data in NSLY_age_earn.csv, the downloaded data set is nsly_rev16. 1. First create nlsy79_rev16_quint_empbins by running nlsy_rev16_create_quint.do. !!!NOTE!!! This requires as input the file all_cutoffs.dta created above using CPS data. 2. Run age_earn_profiles.do README FOR REPRODUCTIONMLE OVERVIEW This portion of the code reproduces a subset of the results: - tables 2&3 (transition and observation matrices) - table 4 (type-specific differences) - appendix tables F.1 and F.2 (parameter estimates) - appendix figures G.1, G.3, and G.4 (model fits and observation bias). It take as given all the data work. ORDER OF OPERATION The master program in this archive is the Matlab program earnproc_030222_dec_final.m. This program calls ancillary Matlab scripts and the dynamic load library functions getLLH_jail_030222_db.mexw64 and getLLH_jail_030222.mexw64, with latter two also being compiled from the *.cpp code. The two versions of the dll differ only in that the db version prints out some intermediate results (debugstuff.txt), while the non-db version does not (for speed). The code loads in data and parameter values from the \data\ subdirectory. Note that you will have to specify the root directory in the *.cpp code, line 457. earnproc_030222_dec_final.m is set up to create debugstuff.txt and then to calculate standard errors (in earnproc030222_dec.txt). Lines 41-43 determine the race x sex x education configuration. The contents of these two output files are then used to create the "transprobs" spreadsheets found in the \results\ subdirectory: for instance, transprobs_bmlhs_LMC_9.xlsx collects results for Black Male Less than High School. Tables 2&3 come from this file, sheet , cells BN54 and BM111. Output from the 16 transprobs spreadsheets is then sorted into earn_coeffs_051022.xlsx (see transprobs sheet ), TypeSumm_051822.xlsx (transprobs sheet , cell BZ250) and the three powerpoint figures (transprobs sheet ). If one uncomments the commented lines in the "iSrch" loop (lines 284-381), earnproc_030222_dec_final.m can be used to estimate parameters. REQUIREMENTS The programs run on a Windows desktop; with 72 processors, the program takes 1-2 minutes to finish. The master program runs on Matlab (R2020b). The Matlab MEX interface is "configured to use 'Microsoft Visual C++ 2019 (C)' for C language compilation". (See https://www.mathworks.com/help/matlab/ref/mex.html for more detail.) Note that the path in line 457 of getLLH_jail_030222_db.cpp and getLLH_jail_030222.cpp will need to be updated. README FOR FOR FORTRAN AND MATLAB RESULTS OVERVIEW This portion of the code reproduces a subset of the results: - the GIRF tables - the age profiles - lifetime summary stats - the counterfactuals It take as given the data work. RUNTIME NOTE The paper uses a simulated panel with 800,000 individuals. On a regular machine (2022 iMac), a single race/sex/ed/permanent type group and setup (e.g., the benchmark) takes around 10 hours to complete, and since there are 32 groups and 10 different setups, the total run time is estimated at 4.5 months. We used a cluster to speed this up by submitting each group/setup as a separate job. To make the code run faster at the expense of some noise, one can replace line 8 of fortran/src/mod_sim.f90 from "integer :: nPanel = 800000" to "integer :: nPanel = 8000", which will speed the code by roughly 100 times resulting in around 12-24 hours of run time. REQUIREMENTS 1. Linux or Mac 2. Matlab 2021a (higher versions probably work) 3. Intel Fortran 19.0.3 or higher with an MPI wrapper (mpif90) and associated MPI libraries. 4. MKL libraries, specifically BLAS and LAPACK and VSL are used 5. A machine with 64gb of memory. A stack size of 500MB per thread is also assumed and may be necessary (if the code segfaults, that is probably the reason). STEPS 1. Put the reproductionFortranMatlab.tar.gz file where you want the results to be saved 2. Extract reproductionFortranMatlab.tar.gz using "tar -xvzf reproductionFortranMatlab.tar.gz" in the terminal (a) If this generates error messages "tar: Ignoring unknown extended header keyword" you can safely ignore those. 3. To reproduce Figure B.1, in matlab/KSYtest/ in Matlab run codes (run time a few seconds) in the following order: (a) testKSY_spline_3(). This produces testspline_3_normal.mat. This should be the same as testspline_3_normal_10bins_0knots.mat up to numerical precision. You can verify this by copying and pasting this into the command window: x = load('testspline_3_normal.mat'); y = load('testspline_3_normal_10bins_0knots.mat'); for fn = fieldnames(x)'; fprintf(1,'Difference for %s is %f\n',fn{1},max(abs(x.(fn{1})-y.(fn{1})),[],'all')); end (b) compare(). This produces the file KSYcompare.eps, which appears in the paper. 4. Go to fortran/ and follow the readme.txt there. Wait for this to complete. You can use "squeue --user= --start" to check the status off the Slurm queued jobs. See the runtime note at the top for the expected runtime. 5. Go to matlab/ and follow the readme.txt there. 5. Finally, inside matlab/forpaper/ are all the graphs and tables produced by the Fortran and Matlab code that get incorporated into the Latex file. NOTE ON RANDOM NUMBER GENERATION The results in the paper were produced with ifort (IFORT) 19.0.3.199 20190206 using GNU Linux Kernel Version #1 SMP Tue Jun 28 15:37:28 UTC 2022 with release 3.10.0-1160.71.1.el7.x86_64 on BigTex. Because random number generation is not reproducible across compilers according to the Fortran standard, I've attempted to use a custom prng that should be reproducible (it uses Intel's MKL library). If somewhere a random number from a call to the default prng slipped in, then the results may vary slightly from machine to machine. However, the simulated sample size is huge. If necessary, we can try to arrange BigTex access for verification. OTHER NOTES The word Stata and Matlab are trademarks of those respective companies. I've used their name in "libgrey" and organized some of the interfaces to be similar to how those programs work, but the code is my own or from others unless otherwise stated.