Menu ENN Search
Language: English Français

DESIGN EFFECT FOR MULTIPLE SMART SURVEY

This question was posted the Assessment forum area and has 18 replies. You can also reply via email – be sure to leave the subject unchanged.

» Post a reply

Stanley Macharia

M&E Officer

Normal user

29 May 2015, 09:40

I need guidance on how to get a single design effect for three separate surveys done last year at Sub-county level. We need to conduct a SMART survey at County level and require a single design effect for sample size calculation. The point estimate we got using guidance provided by Mark Myatt in earlier post.

I will greatly appreciate your help.

Anonymous 81

Public Health Nutritionist

Normal user

31 May 2015, 18:48

This is planning stage. In this case, to be on the save side, I would consider the highest design effect.

Mark Myatt

Consultant Epideomiologist

Frequent user

1 Jun 2015, 05:51

I don't know what "high" means in this context. In eye surveys (e.g.) we might expect to see design effects of seven (7) or higher for Trachoma (an infectious condition that exhibits strong spatial clustering). For nutritional anthropometry surveys like SMART I think that DEFF = 2 will probably be high enough.

I think your question might be about how to calculate the design effect having combined the three surveys. This can be calculated as the ratio of the observed variability to the expected variability assuming a simple random sample. A measure of variability is the width of the 95% confidence interval. An example:

  You combine the three surveys. You have:
  
    n = 1800 (from the three surveys)

  and a estimate of:
  
    p = 12% (95% CI = 9.5%; 14.5%)

  the width of the 95% CI is:
  
    w1 = 14.5% - 9.5% = 5%
    
  We can calculate the expected with of the 95% CI assuming 
  simple random sampling:
  
    n = 1800
    
    p = 0.12 (i.e. 12%)
    
  The width of the 95% CI would be:
    
    2 * 1.96 * sqrt((p * (1 - p)) / n)
    
    2 * 1.96 * sqrt((0.12 * (1 - 0.12)) / 1800) = 0.03 (3%)
    
  The design effect in this example is:
  
    5% / 3% = 1.67

I hope this is of some use.

Anonymous 81

Public Health Nutritionist

Normal user

1 Jun 2015, 07:08

Dear Mark,
Let me explain when I say the highest. As per the information from Anonymous 2744, they are planning to do a survey and I am assuming they want to calculate sample size by estimating expecting prevalence, precision and design effect. Lets me explain how I did say the highest. Lets say the Design effect of the three surveys conducted last year were 1.2, 1.3 and 1.5. My advise was just to take 1.5.

Anonymous 680

Quality Assuarance Officer

Normal user

1 Jun 2015, 07:14

Hi,
Thank you for the reply. But I am not sure how you arrived at the aggregate prevalence.
Thanks

Mark Myatt

Consultant Epideomiologist

Frequent user

1 Jun 2015, 07:18

Yes. That is good advice.

Mark Myatt

Consultant Epideomiologist

Frequent user

1 Jun 2015, 07:28

Sorry to confuse.

The aggregate prevalence was invented. A simple method to arrive at an aggregate prevalence from a set of surveys results is shown in this thread.

Also ... adding to this thread ... It is very common when you have little idea of the design effect needed to specify DEFF=2 unless you have reason to believe that the phenomena of interest if likely to be clustered. Experience with 30-by-30 and SMART surveys suggests that DEFF=1.5 for GAM might be OK. DEFF=2.0 might be safer.

Anonymous 81

Public Health Nutritionist

Normal user

1 Jun 2015, 07:28

I think the aggregated prevalence should be through weighted. My follow up question to anonymous 2744 is on combining of the three surveys. why u did three separate surveys last year in one county? if it was because they are heterogeneous, why you combine now? if they are heterogeneous, u need to have separate unless u need to change the sampling methods.

Frederich Christian Tan

Public Health Practitioner

Normal user

1 Jun 2015, 07:59

Dear anonymous 2744,

You can also look in previous National Surveys conducted (e.g. DHS) as they usually have in their appendix page the design effects (national and sub national level) of all variables used in the survey. This would help you in guestimating your DEFF.

I would also suggest you to get in touch with SMART people. Maybe you can pose your question in their website at http://smartmethodology.org/forums/. I'm sure they are also happy to help you with your concern.


Regards,
Derich

Victoria Sauveplane

Senior Program Manager, Action Against Hunger CA

Normal user

1 Jun 2015, 12:59

To add on to the initial question raised, design effect is a measure of the heterogeneity of your indicator in a given population. That said, it is necessary to consider the context and any aggravating factors when deciding which design effect would be most appropriate.
Generally, if you take a design effect = 2, you are already assuming two different populations with regards to your indicator. For example in the case of malnutrition, you believe that within your population, you expect to see areas with a lot of malnutrition and areas with no malnutrition. Therefore, it is best to stay within the ranges of 1.2 (very homogeneous population) to 1.8 (more heterogeneous population). Remember this is only a 'guestimate' as a colleague has highlighted earlier on. You will be obtaining an issued design effect given your survey data and most often for wasting, it is less than 1.5 (higher design effects are often observed for stunting given the much higher prevalences).

Nevertheless, you still need to consider the context and any aggravating factors. Say you have the following 3 surveys and their observed design effects (DEFF):
Survey A with a DEFF of 1.3
Survey B with a DEFF of 1.14
Survey C with a DEFF of 1.4
This data dates back from 1-2 years and since, more food insecurity has been observed. If you wish to do do 1 survey of all of these areas, then perhaps 1.4 is too low and given the context, 1.7 would now be more appropriate. It is important to discuss and cross-check these assumptions with other country partners as you do not want to unnecessarily inflate your sample size which presents more of a risk of introducing bias.

Hope this helps and I echo Frederich to send your SMART queries to the SMART website or to access any other tools.

Thanks,
Victoria

Anonymous 81

Public Health Nutritionist

Normal user

1 Jun 2015, 13:34

Dear Victoria,
As far as I known, if given communities are heterogeneous, the recommendation has been to have separate surveys. In a given counties or administration area, there could be two distinct livelihoods, say, pastoral, agro-pastoral or agrarian. or could be urban/rural setup. So, in such set up, are you recommending to have single survey just by considering design effect of two?
The other question, in your example, how did you reach 1.7%

Victoria Sauveplane

Senior Program Manager, Action Against Hunger CA

Normal user

1 Jun 2015, 14:24

Apologies for not being clear. Indeed, if you have heterogeneous communities, it is best to do 2 separate surveys (which refers back to my explanation if you assume a design effect of 2 including areas with no malnutrition and those with malnutrition).

Most often, counties or zones of interventions present a mixture of livelihoods and/or urban and rural setups. If you don't have any information on prevalences of your main indicator, then it is best to use 1.5 design effect. If you have data showing differences (say Zone A with 12% GAM and Zone B with 8.5%), and no contextual information presenting major changes in the situation in these areas, then you can use the higher of the two design effects observed for your combined survey of Zone A and B. In the previous example where I said 1.7 should be used, although the other survey data showed that the highest of the 3 surveys had a design effect of 1.4; food insecurity was observed and now we are not sure how that affected our main indicator and it would be better to go with a higher design effect (1.7 as an example) than just using directly 1.4 Remember these are 'guestimates' of what you will obtain with your survey data.

Thanks again,
Victoria

Anonymous 81

Public Health Nutritionist

Normal user

1 Jun 2015, 15:10

Just follow up question regarding increment of design effect when “food insecurity was observed, example of 1.7”. If the situation is deteriorated due to food insecurity across the board (all areas), I am not clear why you recommending to play around gauging the design effect. To me, I would play adjusting around estimated prevalence of by considering the upper level confidence interval.

Victoria Sauveplane

Senior Program Manager, Action Against Hunger CA

Normal user

1 Jun 2015, 15:21

Yes, you would first adjust the prevalence as the starting point of your sample size but also, you would take into account this context and aggravating factors as well when determining what design effect to use.
For further questions regarding sample size calculations using SMART, please see the Sampling Paper on the following link: http://smartmethodology.org/survey-planning-tools/smart-methodology/

Stanley Macharia

M&E Officer

Normal user

2 Jun 2015, 05:23

We had taken 3 separate survey due to different partners having different donors. The donor now is one and there is limit of resources to carry 3 separate surveys.

Mark Myatt

Consultant Epideomiologist

Frequent user

2 Jun 2015, 05:31

Trying to guess at a design effect is (as we can see here) difficult and usually involves a lot of "hand-waving". The time-tested rule-of-thumb is to assume a design effect of 2.0 unless you expect considerable clustering. We have a lot of experience (c. 30 years) with 30-by-30 and SMART surveys (these are related and extremely similar methods) showing the design effect rarely (but sometimes) exceeds 2.0. The older 30-by-30 survey is, perhaps, of some interest here. This was a fixed sample size survey (i.e. n = 900) and no-one worried much about the design effect until data analysis and then it was taken care of by software. It is worth looking at what the 30-by-30 survey did. Estimating a prevalence of 10% with and 95% CI of +/- 3% needs a sample size of n = 384 assuming a simple random sample. The assumed design effect in a 30-by-30 survey was 900/384 = 2.34. This was usually enough to give the expected or better than expected precision.

Anonymous 10423

Normal user

8 Jul 2017, 07:32

Is there a reference for the following:

"If you don't have any information on prevalences of your main indicator, then it is best to use 1.5 design effect."

Bradley A. Woodruff

Self-employed

Technical expert

12 Jul 2017, 18:32

As Mark says, lot's of handwaving and guesstimates. However, there is one paper which presents design effects for various indicators commonly measured in emergency nutrition and health assessment surveys: Kaiser R, Woodruff BA, Bilukha O, Spiegel P, Salama P. Using design effect from previous cluster surveys to guide sample size calculation in emergency settings. Disasters 2006;30:199-211. In the surveys presented in this paper, the design effect for acute malnutrition in children ranges from 0.8 to 2.4, with most surveys falling between 1.1 and 1.6; hence the recommendation for using 1.5 as the assumed design effect.

But design effect is the wrong measure to extract from prior surveys. The design effect is heavily influenced by cluster size, so the design effect from a prior survey with a very different cluster size may not be applicable to your planned survey. The intracluster correlation coefficient (ICC, or sometimes call rho) is a much better measure of the inherent heterogeneity of distribution of an indicator. It reflects the proportion of total variance which is due to differences between clusters. Unfortunately, the editors removed the important discussion of ICC from the paper referenced above, but here is an excellent discussion of ICC: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1466680/.

For this reason, when calculating sample size for a planned survey, I would first decide what the average cluster size will be (perhaps the number of households a team can complete in 1 day), then derive the ICC from prior survey(s). If these prior survey(s) fail to report the ICC (which most survey reports fail to do), but give you the design effect and average cluster size, you can calculate the ICC: ICC = (Design effect - 1) / (Cluster size - 1). Then apply this ICC to the planned survey by calculating the design effect expected in the planned survey using this formula: Design effect = 1 + (Cluster size – 1) x ICC. The process may be overkill for the usual indicators which do not have very high ICCs or design effects, but if you are measuring something for which you expect a lot of heterogeneity of distribution(like water supply, sanitation, vaccination coverage), this step may mean the difference between assuming a design effect of 5 and having wasting survey resources when the actual design effect for your survey data turns out to be only 3.

You can also use the ICC to determine the effect of decreasing the cluster size for the planned survey. If you complete the sample size calculation for an indicator with a high design effect and decide that it is infeasibly large, you can always decrease the cluster size to achieve a lower design effect. However, as shown in the paper cited above, increasing the sample size by increasing the cluster size is an exercise in futility. The increased design effect resulting from increased cluster size usually cancels out any precision advantage from the increased sample size. So if you want to increase the precision of your survey, increase the number of clusters and decrease the size of each cluster.

Mark Myatt

Consultant Epideomiologist

Frequent user

13 Jul 2017, 08:32

Thanks Woody. That covers the issues well.

An earlier article on a related topic is by Binkin et al. (1992). This looks at the 30-by-30 design with PPS selection of cluster locations and proximity sampling within clusters. This, like SMART, is a modified EPI design. A lot of work was done on the EPI design and that provides a rich evidence base covering a large number of child survival indicators.

It is important to realise that the design effect (DEFF) can be altered by design. We can, for example, reduce design effects by having more and / or smaller clusters. DEFF is about accounting for lost variance. A lot of variance is lost when a proximity sample is used. A move to a simple random sample (as in later SMART guidelines) will help to reduce DEFF. A better and simple approach would be to use a within cluster sample design that captures variance by implicit stratification. We can (e.g.) modify the proximity sample to take every third house (EPI3) or every fifth house (EPI5). We can also split the sample by taking a small samples from different parts of a sampled community (segmentation) - these can be thought of as spatially selected clusters within clusters. RAM and S3M samples often use a combination of segmentation and EPI3/EPI5 for within cluster sampling.

These sorts of modifications to within clusters sampling methods can go a long way to improving the statistical efficiency of cluster samples while maintaining their cost efficiency. They are not a panacea as variance loss is can be due to the cluster selection methods. For example, PPS will tend to select larger communities making it a poor choice for some indicators. Spatial stratification can help with this. The main thing is to have a large enough sample of clusters. A good minimum for a SMART type survey is about 30 clusters. A good minimum for a survey using a spatial sample of clusters and segmentation with EPI3/EPI5 in clusters is 16 - since we typically take 3 or 4 sub-clusters from within each cluster this give about 48 to 64 very small clusters.

I hope this is of some use to somebody.

Back to top

» Post a reply