Dear Professor, Could you please confirm that: For the countries you provided, are their row counts complete? Are you sure that these countries only have this exact number of rows?

Thanks!

I think sample2.tsv is just an example to demonstrate the output format. Obviously there should be more data if you are cleaning from the original files.

ANONYMOUS wrote:

Dear Professor, Could you please confirm that: For the countries you provided, are their row counts complete? Are you sure that these countries only have this exact number of rows?

Thanks!

Hi I certainly believe so. Don't forget, you have all the TSV source files, so you can work forward and see what should be there. Have I made a mistake?

Cheers MichaelW

Hi Michael

Could I do a sanity check?

I believe I have filtered the tsv data per requirements and about the exclude rows that do have insufficient Cantril-Value pairs. This will be done on the basis that:

• Cantril data may be absent in certain years within the range of those for which there otherwise is data - those cells should be retained;
• Empty cells are fine anywhere for the data cleaning program, so long as the rows are within the range of years of interest (thread with Kai Zheng at 3.37pm, 19 May);and
• Cantril data could be missing for some countries, wholly or in part. If so, to include a correlation for any country there be at least 3 predictor-value, Cantril-value pairs.

Looking at my filtered data against Sample 2, I have four rows that, to my understanding, should be retained. Please find my data in attached tsv.

My thought process is as follows: Afghanistan 2013 doesn't have a Canrtil Ladder Score and should be discarded.

Afghanistan 2014 and UAE 2016-2018 should be retained as they contain three predictor value pairs as well as the Cantril Ladder Score.

Or are those four rows excluded because the entire row/value set is incomplete? The countries however are included in the correlation because they have at lease 3 years of full row/value sets with a Cantril Ladder Score?

Thank you.

Apologies. I tried to attach the tsv and it failed. Hope this works

#Entity	Code	Year	GDP per capita, PPP (constant 2017 international $)	Population (historical estimates)	Homicide rate per 100,000 population - Both sexes - All ages	Life expectancy - Sex: all - Age: at birth - Variant: estimates	Cantril ladderscore
Afghanistan	AFG	2011	1961.0963	29249156	4.208668	61.4	4.25835
Afghanistan	AFG	2012	2122.8308	30466484	6.393913	61.9	
Afghanistan	AFG	2013	2165.3408	31541216		62.4	
Afghanistan	AFG	2014	2144.4497	32716214		62.5	3.575
Afghanistan	AFG	2015	2108.714	33753500	9.975262	62.7	3.36
Afghanistan	AFG	2016	2101.422	34636212	6.6924186	63.1	3.794
Afghanistan	AFG	2017	2096.093	35643420	6.8006945	63	3.6315
Afghanistan	AFG	2018	2060.699	36686788	6.7435727	63.1	3.2033
Afghanistan	AFG	2019	2079.9219	37769496	7.180397	63.6	2.5669
Afghanistan	AFG	2020	1968.341	38972236	6.594439	62.6	2.5229
Afghanistan	AFG	2021	1516.3057	40099460	4.0224977	62	2.4038
United Arab Emirates	ARE	2011	57815.17	8575210	0.59473795	78.5	6.977243
United Arab Emirates	ARE	2012	59949.246	8664976	0.79630977	78.7	
United Arab Emirates	ARE	2013	62354.824	8751853	0.6512911	78.9	
United Arab Emirates	ARE	2014	64334.09	8835957	0.69036144	79	6.901
United Arab Emirates	ARE	2015	68076.63	8916909	0.67287964	79.2	6.573
United Arab Emirates	ARE	2016	71244.586	8994266		79.3	6.648
United Arab Emirates	ARE	2017	71182.37	9068297		79.5	6.7741
United Arab Emirates	ARE	2018	71550.555	9140172		79.6	6.8245
United Arab Emirates	ARE	2019	71782.16	9211660	0.6947718	79.7	6.7908
United Arab Emirates	ARE	2020	67668.29	9287286	0.6998813	78.9	6.5605
United Arab Emirates	ARE	2021	69733.8	9365149	0.46982723	78.7	6.576
Bahamas	BHS	2011	34468.887	377956	33.60233	72.6	
Bahamas	BHS	2012	35150.562	382073	28.791214	72.8	
Bahamas	BHS	2013	33826.34	385660	30.856995	73	
Bahamas	BHS	2014	34143.03	389137	31.35191	73.4	
Bahamas	BHS	2015	34170.23	392707	37.178745	73.1	
Bahamas	BHS	2016	33596.637	395986	28.031967	73.5	
Bahamas	BHS	2017	34357.27	399027	30.57491	73.6	
Bahamas	BHS	2018	34735.125	401911	22.642082	73.8	
Bahamas	BHS	2019	35161.832	404563	23.482475	71.2	
Bahamas	BHS	2020	26659.238	406478	17.959438	72.7	
Bahamas	BHS	2021	30210.162	407920	29.173424	71.6	
Oman	OMN	2011	37745.453	3206883	1.0602238	76.6	
Oman	OMN	2012	37270.586	3535585	1.1030726	77.1	
Oman	OMN	2013	36330.48	3816688	2.5414758	77.2	
Oman	OMN	2014	35032.258	4009272	0.29930654	77.4	6.853
Oman	OMN	2015	35188.023	4191784	0.38169974	77.7	
Oman	OMN	2016	35229.953	4398080	0.36379594	77.9	
Oman	OMN	2017	34218.387	4541853	0.3742965	77.9	
Oman	OMN	2018	34212.105	4601160	0.28253767	78	
Oman	OMN	2019	33814.113	4602769	0.49969932	78	
Oman	OMN	2020	33098.21	4543406	0.30813938	74.8	
Oman	OMN	2021	34294.766	4520474	0.24333748	72.5

I guess he was wondering whether we should calculate the country that lacks some column. Like some rows have data on GDP, but not homicide. Should we include a valid column in our correlation or just ignore the entire line.