thesis /
07-Chp-Triangulation.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661---
bibliography: bibliography/references.bib
csl: bibliography/nature.csl
output:
bookdown::pdf_document2:
template: templates/brief_template.tex
bookdown::word_document2:
toc: false
toc_depth: 3
reference_docx: templates/word-styles-reference-01.docx
number_sections: false
bookdown::html_document2: default
documentclass: book
---
```{block type='savequote', include=knitr::is_latex_output(), quote_author='(ref:tri-quote)', echo = TRUE}
Next on my list of features to be specially considered I would place the consistency of the observed association. Has it been repeatedly observed by different persons, in different places, circumstances and times?
```
(ref:tri-quote) --- Austin Bradford Hill, 1965[@hill1965]
<!-- IDEA Consideration of the potential impact of the magnitude and direct of residual confounders/bias is not a major stretch from what is already happening in the assessment of the quality of evidence (GRADE) framework. Within GRADE, the overall quality of evidence can be upgraded when there is deemed to be unmeasured or residual confounding variables which reduce the. For example, if the propensity to treatment is related to comorbidity burden, but those on treatment still have better outcomes then those on control, it is likely that the true effect of the intervention is being underestimated. [@guyatt2011] -->
# Aetiological triangulation across new and existing evidence sources: a qualitative and quantitative approach {#tri-heading}
\minitoc <!-- this will include a mini table of contents-->
```{r, echo = FALSE, warning=FALSE, message=FALSE}
source("R/helper.R")
knitr::read_chunk("R/03-Code-Systematic-Review.R")
knitr::read_chunk("R/05-Code-CPRD-Analysis.R")
knitr::read_chunk("R/06-Code-IPD-Analysis.R")
knitr::read_chunk("R/07-Code-Triangulation-Analysis.R")
doc_type <- knitr::opts_knit$get('rmarkdown.pandoc.to') # Info on knitting format
```
\newpage
::: {.laybox data-latex=""}
## Lay summary {-}
Sources of evidence often contain limitations which reduce our confidence in their results. To minimise the impact of these limitations, a method has been developed called triangulation which looks at multiple sources addressing the same question. If the result from each source points towards the same answer, this improves our confidence in the conclusion.
In this chapter, I describe a new approach for triangulation, which combines two existing methods. This approach is used to ensure that the results meet the following criteria:
* robustness - do we believe the result, based on the methods/results of the study?
* relevance - does the source provide useful evidence relevant to the question we are interested in?
This triangulation approach is quantitative in nature, meaning I have tried to combine the results from different sources mathematically to obtain one single overall result. I have applied this method to assess the following relationships of interest:
* the effect of midlife LDL-c on Alsheimer’s disease
* the effect of midlife triglycerides on vascular dementia
My method did not support the existence of an effect in either relationship of interest. I close the chapter by suggesting future work to further develop and improve upon the new method presented.
:::
<!----------------------------------------------------------------------->
<!-- TODO Need to define exposures and acronyms early so that I can use abbrev in results -->
## Introduction {#triangulation-overview}
Aetiological triangulation, or simply triangulation, is the process of comparing and contrasting across different sources of evidence. Triangulation is broadly comparable to Bradford-Hill's criteria of "consistency", that is the replication of an observed relationship across several different contexts,[@hill1965] where contexts are assumed to have different underlying bias structures. More formally it can be defined as:
> The practice of strengthening causal inferences by integrating results from several different approaches, where each approach has different (and assumed to be largely unrelated) key sources of potential bias.[@lawlor2016]
This approach represents a significant step forward from the current practice of synthesising evidence from only one type of study (e.g., randomised controlled trials (RCTs), non-randomised studies of exposures (NRSE) or interventions (NRSI)). The most common implementation of this method to date has been in the form of _qualitative triangulation_ - that is the identification and narrative comparison of diverse evidence sources with respect to their underlying bias structures.[@lawlor2016]
However, qualitative triangulation faces issues at scale. There is a need to include all evidence to avoid potential confirmation bias,[@dubroff2018] yet as the number of evidence sources increases, it is not possible to narratively compare and contrast across multiple results in any meaningful way. This fact is illustrated by previous exemplars of triangulation only considering a small number of individual results.[@ference2014; @lawlor2016] To overcome this limitation, some attempts have considered the output of a meta-analysis of similarly designed studies to be a single source of evidence, and have assessed bias in relation to this summary result. While this approach keeps the number of individual sources of evidence manageable, it loses useful information on the specific biases inherent to each result contributing to the summary estimate. For example, while it may be true that all non-randomised studies share some minimal level of bias, competing extents and directions of bias in individual results may be masked by the use of summary estimates. In addition, previous attempts at qualitative triangulation have not systematically assessed the indirectness of the results, namely the differences between the question of interest and the question addressed by a study.
Taken together, these limitations to qualitative triangulation detail a need for a more systematic way to integrate across multiple evidence sources as the number of individual results contributing to the exercise increases. One such approach is _quantitative triangulation_, where results from different evidence sources are combined numerically while accounting for the key sources of bias and indirectness in each. This chapter, in addition to providing a narrative comparison of the existing and new evidence identified by this thesis, builds on recent developments in risk-of-bias assessment and methods for bias-/indirectness-adjusted meta-analysis to propose a generalised framework for quantitative triangulation. The framework, along with the methodological challenges to its implementation, is illustrated via two case studies examining the causal effect of lipids on dementia outcomes.
<!----------------------------------------------------------------------->
## Data sources {#tri-data-sources}
The triangulation exercises presented in this chapter draw on the research produced in each of the preceding chapters. More specifically, this chapter builds on the comprehensive systematic review of existing evidence presented in Chapters \@ref(sys-rev-methods-heading) & \@ref(sys-rev-results-heading). This evidence base is then supplemented by new evidence on the association of statin use with dementia outcomes in the CPRD (Chapter \@ref(cprd-analysis-heading)) and the association of lipids (namely, total cholesterol (TC), high-density lipoprotein cholesterol (HDL), low-density lipoprotein cholesterol (LDL) and triglycerides (TG)) with dementia outcomes in previously unanalysed datasets accessed via the DPUK (Chapter \@ref(ipd-heading)).
Table \@ref(tab:thesisOverview-table) summarises the research question each data source has attempted to address, the exposures and outcomes considered in each case, and the contribution of each chapter to the triangulation exercise presented here.
\blandscape{}
<!----------------------------------------------------------------------------->
(ref:thesisOverview-caption) __Summary of research designs included in this thesis__ -The research projects performed as part of this thesis and used as evidence sources in this triangulation exercise are summarised. Note that Chapter \@ref(sys-rev-tools-heading) is intentionally not included in this table, as it describes a research tool rather than a research study.
(ref:thesisOverview-scaption) Summary of research designs included in this thesis
```{r thesisOverview-table, message=FALSE, results="asis", echo = FALSE}
```
<!----------------------------------------------------------------------------->
\elandscape{}
Additionally, both the qualitative and quantitative triangulation exercises used the detailed risk-of-bias assessments performed and presented as part of the systematic review (Chapters \@ref(sys-rev-methods-heading) & \@ref(sys-rev-results-heading)). Similarly, risk-of-bias assessments were performed for each new source of evidence presented in this thesis. The risk of bias tools used to assess each result are described in detail in Section \@ref(risk-of-bias), but in summary, randomised controlled trials (RCTs) were assessed using the RoB 2 tool;[@sterne2019] non-randomised studies of interventions (NRSI) using the ROBINS-I tool;[@sterne2016] non-randomised studies of exposures (NRSE) using an early adapted version of the ROBINS-E tool (see Section \@ref(rob-tools-nrse));[@morganr2020; @french2019] and Mendelian randomisation (MR) studies using the Mamluk et al. tool.[@mamluk2020]
<!----------------------------------------------------------------------->
## Qualitative (narrative) triangulation {#qual-tri}
```{r rctStatinDementia, include = FALSE, message=FALSE, warning=FALSE}
```
```{r Hypercholesterolemia, include = FALSE, message=FALSE, warning=FALSE}
```
```{r mrLipidsAD, include = FALSE, message=FALSE, warning=FALSE}
```
```{r LipidsSD, include = FALSE, message=FALSE, warning=FALSE}
```
```{r lipidFrationsSetup, include = FALSE, message=FALSE, warning=FALSE}
```
```{r obsStatins, include = FALSE, message=FALSE, warning=FALSE}
```
```{r obsFibrates, include = FALSE, message=FALSE, warning=FALSE}
```
```{r mrStatins, include = FALSE, message=FALSE, warning=FALSE}
```
```{r azd-text, echo = FALSE}
```
```{r prepIPDFigures,include=F}
```
As part of a qualitative triangulation of the new and existing evidence identified by this thesis, all information sources were initially grouped by outcome, and the findings from each source were narratively compared and contrasted. Potential reasons for heterogeneity between study designs were examined with specific reference to the risk-of-bias in each.
<!----------------------------------------------------------------------->
#### All-cause dementia
Analysis of lipid levels across both the systematic review (Chapters \@ref(sys-rev-methods-heading) & \@ref(sys-rev-results-heading)) and the individual participant data (IPD) analysis (Chapter \@ref(ipd-heading)) found extremely weak evidence for an association of any lipid fraction with all-cause dementia (in the systematic review: TC - `r obsLipids$Dementia_TC$estimate`; [LDL-c]{.correction} - `r obsLipids$Dementia_LDL$estimate`; [HDL-c]{.correction}
- `r obsLipids$Dementia_HDL$estimate`; TG - `r obsLipids$Dementia_TG$estimate` ). Similarly, meta-analysis of studies identified by the review which examined a binary hypercholesterolemia provided weak evidence for an association of this risk factor with Alzheimer's disease (`r obsHyperchol$Dementia$estimate`). Finally, a meta-analysis of RCTs of statin use (`r rct_statin_acd`) and of MR analyses of genetic instruments that mimic statin use (`r mrStatin$Dementia$estimate`) found weak evidence of an association, providing indirect evidence on the effect of lipid lowering on this outcome.
In contrast, the meta-analysis of statin use found a slight protective effect for the all-cause dementia outcome (`r obsStatins$Dementia$estimate`), although many of the studies in this analysis were found to be at risk of immortal time bias, which would be expected to favour the statin use. Finally, analysis of the association of statin use with all-cause dementia in the CPRD (Chapter \@ref(cprd-analysis-heading)) found evidence of a harmful association (`r anydem_sta_text`). However, this finding is likely to be driven by the substantial confounding by indication identified in the vascular dementia subgroup.
For fibrates, an alternative lipid-regulating agent, there was some evidence of an increased risk of all-cause dementia in the CPRD (`r anydem_fib_text`), though this was not consistent in the meta-analysis of previous studies examining the association of fibrates and all-cause dementia (`r obsFibrates$Dementia$estimate`).
<!----------------------------------------------------------------------->
#### Alzheimer's disease
Meta-analysis of lipid levels in the systematic review found extremely weak evidence for an association of any lipid fraction with Alzheimer's disease (in the systematic review: TC - `r obsLipids$AD_TC$estimate`; [LDL-c]{.correction} - `r obsLipids$AD_LDL$estimate`; [HDL-c]{.correction} - `r obsLipids$AD_HDL$estimate`; TG - `r obsLipids$AD_TG$estimate`). Similarly, meta-analysis of studies examining a binary hypercholesterolemia provided weak evidence for an association of this risk factor with Alzheimer's disease (`r obsHyperchol$AD$estimate`). A meta-analysis of MR analyses of genetic instruments that mimic statin use found weak evidence for an association of lipid lowering with Alzheimer's disease (`r mrStatin$AD$estimate`).
In contrast, the meta-analysis of statin use found a slight protective effect on this outcome (`r obsStatins$AD$estimate`), though similar to the all-cause dementia outcome, many of the studies in this analysis were found to be at risk of immortal time bias. However, the CPRD analysis, which attempted to account for this potential bias through the use of time-varying treatment indicator, found weak evidence for an effect (`r probad_sta_text`). The validity of this finding was limited by the potential for differential misclassification on the basis of the exposure in the CPRD analysis.
Finally, only one of the three cohorts included in the IPD analysis (Whitehall II) contained data on Alzheimer's disease. However, a previously published analysis (Tynkkynen _et al._[@tynkkynen2018]) examining the association of lipid levels with Alzheimer's disease in this cohort was identified by the systematic review. The results of this previous analysis are incorporated into the results of the meta-analysis reported above, and so the IPD analysis provided no new information on this outcome.
<!----------------------------------------------------------------------->
#### Vascular dementia
There was a general absence of existing evidence on vascular dementia outcomes. Meta-analysis of published results was only possible for the total cholesterol lipid fraction (`r obsLipids$VaD_TC$estimate`). A meta-analysis of studies examining hypercholesterolemia also found weak evidence for an association with vascular dementia (`r obsHyperchol$VaD$estimate`). Similarly, a meta-analysis of statin use found weak evidence for an effect on this outcome (`r obsStatins$VaD$estimate`). The CPRD analysis found evidence for a harmful association of statin use (`r vasdem_sta_text`), though this finding is likely to be the result of severe confounding by indication related to vascular factors, as identified through the use of control outcomes.
The IPD analysis provided supporting evidence for the absence of an effect of any fraction on vascular dementia, with the sole exception of triglycerides. Raised triglycerides were associated with an increased incidence of vascular dementia across two previously unanalysed cohorts (`r ipd_vasdem`), although there is the potential for uncontrolled confounding due to the limited information available on important covariates. With respect to evidence on treatments for hypertriglyceridemia, examination of fibrates in the CPRD provided weak evidence of an association with vascular dementia (`r vasdem_fib_text`).
<!----------------------------------------------------------------------->
## Quantitative triangulation
The qualitative comparison presented above identified no consistent effect of any lipid fraction on any dementia outcome across the evidence sources presented in this thesis. However, it is clear that this qualitative approach faces difficulties in interpretation when the number of individual results available is large, as discussed in the introduction to this chapter. This is true even when using summary estimates from meta-analyses of primary studies, which sacrifice valuable information on the biases inherent to each individual result.
In the following section, I propose and apply a novel quantitative triangulation framework to address these limitations. This approach incorporates recent advancements in the way that bias in results is assessed (namely the move to domain-based assessment tools)[@sterne2019] and existing methods for bias-/indirectness-adjusted meta-analysis[@turner2009] to integrate the numerical results of the multiple approaches. Despite the absence of any clear signals across the evidence base, as detailed in the previous section, I have used the evidence identified and produced by this thesis to illustrate the novel methodological approach.
<!----------------------------------------------------------------------->
### Methods
The proposed framework involves the following steps:
1. Define the causal question of interest
1. Identify relevant evidence sources and standardise effect directions
1. Specify an idealised version of each study
1. Assess the extent and direction of bias/indirectness in each result
1. Define modifying terms for bias and indirectness in each result
1. Calculate bias-/indirectness-adjusted results and perform meta-analysis
Each step is described in comprehensive detail in the sections below. To better illustrate the bias-/indirectness adjustment process detailed in the subsequent sections, the process for calculating the adjusted estimate is described in detail for a single result. Following this, the framework is then applied to two case studies.
<!----------------------------------------------------------------------->
#### Definition of the causal questions of interest (case-studies)
Using the ROBINS-E framework,[@morganr2020] I defined the parameters of my causal questions of interest:
_Case study #1: Effect of mid-life LDL-c on Alzheimer's disease_
* _Population of interest_: General population
* _Exposure of interest_: Low density lipoprotein cholesterol
* _Exposure window of interest_: Mid-life (45-60)
* _Summary measure of exposure over time_: Average exposure
* _Outcome of interest:_ Alzheimer's disease
_Case study #2: Effect of mid-life triglycerides on vascular dementia_
* _Population of interest_: General population
* _Exposure of interest_: Triglycerides
* _Exposure window of interest_: Mid-life (45-60)
* _Summary measure of exposure over time_: Average exposure
* _Outcome of interest:_ Vascular dementia
<!-- QUESTION Should population here include a geographical aspect, as in the Discussion I say that the generalisability to other populations is an issue. -->
These causal questions were chosen for several reasons. Firstly, the work presented in the previous chapters had identified some evidence for an association in these exposure/outcome pairs. Secondly, given the long prodromal period between the onset of physiological changes in the brain and presentation of dementia symptoms, it seems likely that conditions during the mid-life period are particularly important. Finally, examining the causal impact of a lipid fraction on a specific outcome (e.g., Alzheimer's disease), rather than a composite outcome (e.g., all-cause dementia), focuses on a single causal pathway rather than including several conditions in the outcome definition that likely have very different mechanisms of disease. In addition, previous work presented in this thesis illustrated that the component conditions (in this case Alzheimer's disease and vascular dementia) are likely to be subject to very different confounding structures (see Section \@ref(cprd-confounding-by-ind)), and the use of a composite all-cause outcome in the triangulation framework may mask this.
<!----------------------------------------------------------------------->
#### Identify relevant results
Once the causal question of interest had been defined, relevant individual results were obtained from the data sources described in Section \@ref(tri-data-sources). In a broad sense, this meant extracting results that examined the relationship between the exposure/outcome pair, either directly (e.g., non-randomised studies of LDL-c levels) or indirectly (e.g., non-randomised studies of statins, or MR studies using genetic instruments for LDL-c levels).
Effects were standardised to refer to the risk resulting from a "reduction" in the lipid fraction. For both non-randomised studies and Mendelian randomisation analysis of lipid levels, the association is usually reported in terms of an increase in the lipid fraction. In this case, the effect estimates were inverted to ensure consistency across study designs.
<!-- TODO Note that the results are converted to the log scale. -->
<!-- TODO The results should be standarised before calculating direction of effect, because other-wise it is pulling it in the wrong direction - YOU NEED TO SAY THIS LATER! -->
<!-- REVIEW How did you deal with the statin studies? Did you adjust for the average reduction in LDL-c for average dose? Need to add clarification here.
This should be an issue, as different effect sof statins will be adjusted for bia indirectness aspect of framework.
Move to questions
-->
<!----------------------------------------------------------------------->
#### Specify idealised version of each study
Once the set of relevant results have been identified, an idealised study for each should be described. An idealised study can be viewed as a replicate of the study producing the result of interest but with all sources of bias removed. The risk of bias is then assessed against this idealised study, while the causal question addressed by each idealised study is compared with the target causal question to define the indirectness of each result.
Note that all of the risk-of-bias tools used to assess non-randomised studies required specification of an idealised version of the study, against which bias was assessed. If using a risk-of-bias assessment tool that does not require this step, the idealised version of the study should be defined in advance of risk-of-bias assessment.
<!----------------------------------------------------------------------->
#### Assess risk and direction of bias in each result
The assessment of bias in each result is discussed in Section \@ref(tri-data-sources) above. For the sake of consistency and ease of computing, I mapped the different acceptable risk-of-bias judgements in each tool to a harmonised set: "Low", "Moderate", and "High". The exact mapping can be seen in Table \@ref(tab:robLevelsMapping-table). Of note, no mapping was performed for the critical risk-of-bias levels present in the tools for non-randomised studies. This is in accordance with current best practice in evidence synthesis, which is to exclude all studies at critical risk of bias from further quantitative synthesis.[@sterne2016]
<!----------------------------------------------------------------------->
(ref:robLevelsMapping-caption) __Mapping of risk-of-bias judgements across assessment tools__ - The different acceptable levels of bias across the risk-of-bias assessment tools used were mapped to a harmonised set. Note that no mapping for "Critical" risk of bias was performed, as best practice recommends that studies at critical risk of bias are excluded from quantitative syntheses.
(ref:robLevelsMapping-scaption) Mapping of risk-of-bias judgements across assessment tools
```{r robLevelsMapping-table, message=FALSE, results="asis", echo = FALSE}
```
<!----------------------------------------------------------------------->
In addition to assessing the extent of bias, as defined by the three levels detailed above, I recorded the predicted direction and type of bias in each domain. This was achieved via an additional question at the end of each bias domain, which asked assessors to predict the expected direction of bias in that domain. Note that this question is currently only formally included in two existing tools (ROB2/ROBINS-I), but I employed an identical approach during assessment of non-randomised studies of exposure and Mendelian randomisation studies. Acceptable responses to this question included:
<!-- REVIEW in fact it is a mandatory part of new ROBINS-E tool, while only optional in RoB2 and ROBINS-I. Maybe reiterate that using a very early version of ROBINS-E -->
* "Favours experimental"/"Favours comparator" (defines an additive bias)
* "Towards null"/"Away from null" (defines a proportional bias)
* "Unpredictable"
<!-- REVIEW Is it worth explaining the terms experimental and comparator above? -->
As indicated in the options above, the response to this question is also used to determine the predicted type of bias as either additive or proportional. Proportional biases are dependent on the magnitude of the effect, while additive biases are not.[@turner2009]
<!-- TODO Need to include some text here on examples of each: e.g., immortal time bias, or -->
For additive biases, determining the absolute direction of bias was simply a case of whether the bias is expected to shift the effect estimate to the left or right on an imaginary forest plot. For example, the estimate would be shifted to the right if a bias was predicted to favour the comparator (Figure \@ref(fig:exampleDirection)).
In contrast, for proportional biases, whether the estimate should be increased or decreased proportionally depends on the current position of the point estimate relative to the null. For example, if the effect estimate represents a protective effect (below the null), then bias towards the null would be adjusted for by moving the effect estimate proportionally to the right. In contrast, if the effect of the intervention is harmful (effect estimate above the null), bias towards the null would be adjusted for by moving the effect estimate proportionally to the left. Both scenarios are illustrated using example data in Figure \@ref(fig:exampleDirection).
<!----------------------------------------------------------------------------->
```{r exampleDirectionSetup, include = F}
```
(ref:exampleDirection-cap) __Calculating the absolute direction of biases, based on bias type__ - For additive biases, the direction is consistent regardless of the position of the effect estimate relative to the null - bias "favouring the comparator" will always pull the effect estimate to the right and can induce a change from a protective to harmful effect. In contrast, the position of the effect estimate is accounted for when determining the absolute direction of proportional biases, defined as towards or away from the null.
(ref:exampleDirection-scap) Calculating the absolute direction of biases, based on bias type
```{r exampleDirection, echo = FALSE, results="asis", fig.pos = "H", fig.cap='(ref:exampleDirection-cap)', out.width='100%', fig.scap='(ref:exampleDirection-scap)'}
knitr::include_graphics(file.path("figures/tri/exampleDirection.png"))
```
<!----------------------------------------------------------------------------->
The predicted direction and type of bias was not recorded for domains at "Low" risk of bias. Finally, for some domains, only one type of bias is allowed. For example, in the confounding domains in the ROBINS-I/ROBINS-E tools, only the "Favours experimental"/"Favours comparator" and "Unpredictable" options are available, meaning that bias in this domain will always be considered additive.
<!----------------------------------------------------------------------->
#### Visualisation of extent and direction of bias
To aid with the quantitative triangulation exercise, I created a new method to visualise the extent, direction and type of bias in each result, enabling detailed comparison across different studies contributing to a synthesis. Similar to the standard paired forest plots introduced in Chapter \@ref(sys-rev-methods-heading), these "bias direction" plots show the level (or extent) of bias across the domains for each result reported using coloured blocks. However, in contrast to the previous forest plots, symbols are used to illustrate the predicted absolute direction of bias in that domain. Distinct symbol pairs are used to denote additive and proportional biases. Illustrations of these are provided in the Section \@ref(quant-tri-results).
<!----------------------------------------------------------------------->
#### Assign modifying values to risk/direction of bias {#tri-bias-assign}
<!-- TODO Note that they are on the log scale, and that this requires the results to be on the log scale too! - Lines 45-50 of addbias -->
Following definition of the extent, type and direction of bias in each domain, I assigned a prior distribution on the log scale to each combination. For example, a "high" additive bias in $j$th domain of the $i$th study is defined as $\delta_{i,j}^{\mathrm{High}}$ and uncertainty around this estimate is given by a distribution with mean $\mu$ and variance $\sigma^{2}$:
\begin{equation}
\delta_{i,j}^{\mathrm{High}} = f(\mu_{i,j}^{\mathrm{High}}, {\sigma_{i,j}^{\mathrm{High}}}^{2})
(\#eq:additive-single)
\end{equation}
The sign of $\mu_{i,j}^{\mathrm{High}}$ is defined by the absolute direction of the bias. If the bias is expected to pull the effect to the left, $\mu_{i,j}^{\mathrm{High}}$ is given a negative sign, and if it is expected to pull the effect to the right, it is given a positive sign.
<!-- QUESTION sign is negative if to the left, because you are taking it away in the final formula and it therefore needs a negative sign to be positive overall! -->
<!-- TODO Need to explain here that for proportional bias, the results are exponentiated -->
One key feature of the domain-based risk-of-bias tools is that the domains are considered interchangeable - i.e., a "high" judgement in one domain is equivalent to a "high" judgement in any other. As such, for this analysis, I assumed that the distributions assigned to each extent of bias were identical across all values of $j$, that is, were identical for all domains in the risk-of-bias assessment tool. To illustrate for additive biases:
$$
\delta_{i,1}^{\mathrm{High}} = \delta_{i,2}^{\mathrm{High}} = ... = \delta_{i,j}^{\mathrm{High}}
$$
and similarly
$$
\delta_{i,1}^{\mathrm{Moderate}} = \delta_{i,2}^{\mathrm{Moderate}} = ... = \delta_{i,j}^{\mathrm{Moderate}}
$$
Reasonable values for the position and spread of the prior distributions were informed by a previously reported expert bias elicitation exercise.[@turner2009] Using open data from that study, I calculated the average values for the mean and variance of the adjustment distributions across all biases considered by that exercise (see Appendix \@ref(appendix-adjustment-values) for more details on how these were calculated). For the purposes of this illustrative analysis and to highlight the generalised nature of the code used to perform the analysis (see Section \@ref(tri-software)), I used the calculated values to inform two scenarios of modifying distributions (Table \@ref(tab:priorsAdd-table)) for additive biases, and a single set for proportional biases. In the first scenario, the relationship between the position assigned to each additive bias judgement is linear. In the second scenario, the step from "Moderate" to "High" is twice that from "Low" to "Moderate". Across both additive and proportional biases, the variance of the "High" judgement distribution was defined to be greater than that of the "Moderate" judgement.
<!----------------------------------------------------------------------->
(ref:priorsAdd-caption) __Prior distributions mapped to different extents of bias__ - Prior values assigned to additive and proportional biases, defined as log-normal distributions with mean $\mu$ and variance $\sigma^2$ using the notation $N(\mu,\sigma^{2})$. For additive biases, two scenarios were defined (see main text). Note that proportional biases are defined on the log scale for ease of calculation but are exponentiated prior to their use for adjustment (see Equations \@ref(eq:prop-total-mean) & \@ref(eq:prop-total-var) below).
(ref:priorsAdd-scaption) Prior distributions mapped to different extents of bias.
```{r priorsAdd-table, message=FALSE, results="asis", echo = FALSE}
```
<!----------------------------------------------------------------------->
<!-- TODO In the absence of any empirical data regarding the priors, I used the data from the expert elicitation approach to calculate both the mean and max. These estimates, in addition to conversations with topic experts -->
<!-- TODO Link this to open data! -->
Where the direction of bias in a domain was unpredictable for a given result, the mean of the prior distribution for that domain was set to 0 but the appropriate variance for the recorded extent of bias was retained (e.g., for an unpredictable "High" additive bias, the distribution would be $N(0,0.1)$ under Scenario 1). In other words, adjusting for bias in a domain with an unpredictable direction of bias has no impact on the position of the effect estimate but does increase the uncertainty around it.
Under the method reported in Turner _et al._,[@turner2009], total additive bias (denoted using subscript $\delta$) in result $i$ is then described by mean $\mu_{i\delta}$ and variance $\sigma_{i\delta}^{2}$, which are simply the sum of the mean and variance of all additive biases in the $i$th result. Similarly, the total proportional bias (denoted using subscript $\beta$) in the $i$th result is described by mean $\mu_{i\beta}$ and variance ${\sigma_{i\beta}}^2$. This are defined as
```{block type='correction'}
\begin{equation}
\mu_{i\beta} = exp(\alpha_{i} + {\psi_i}^2/2)
(\#eq:prop-total-mean)
\end{equation}
```
\begin{equation}
{\sigma_{i\beta}}^2 = \big\{ exp({\psi_{i}}^2)-1 \big\} exp({2}{\alpha_{i}} + {\psi_{i}}^2)
(\#eq:prop-total-var)
\end{equation}
where $\alpha_i$ and ${\psi_i}^2$ are the sum of mean and variance values for all proportional biases in the $i$th result.
Note that the total additive bias is maintained on the log scale, while the total proportional bias has been exponentiated.
<!----------------------------------------------------------------------->
#### Assessing and assigning prior values to indirectness
The indirectness of a result (also termed external bias, external validity, relevance, generalisability, or applicability) is defined here as the discrepancy between the research question addressed by the "idealised" study and the causal question of interest.
An identical approach was employed to assess, and assign prior distributions to, the indirectness in each result. The target question for each result was compared against the causal question of interest with respect to three domains, defined as important by the GRADE framework for assessing the certainty of evidence:[@guyatt2011] population, intervention/exposure, and outcome. I again used the scale of "Low/"Moderate"/"High" to quantify the extent of indirectness in each domain. All indirectness was defined _a priori_ as being proportional in nature (i.e. depending on the magnitude of the effect), in line with previous work on this topic.[@turner2009;@thompson2011] As above, data from the previous elicitation exercise were used to inform reasonable prior distributions for each level of indirectness (Table \@ref(tab:priorsIndirect-table)).
<!----------------------------------------------------------------------------->
(ref:priorsIndirect-caption) __Prior distributions mapped to different extents of indirectness__ - Prior values assigned to proportional indirectness, defined as log-normal distributions with mean $\mu$ and variance $\sigma^2$ using the notation $N(\mu,\sigma^{2})$. Note that all indirectness in this analysis was defined _a priori_ to be proportional, and so no distributions for additive indirectness were defined.
(ref:priorsIndirect-scaption) Prior distributions mapped to different extents of indirectness.
```{r priorsIndirect-table, message=FALSE, results="asis", echo = FALSE}
```
<!----------------------------------------------------------------------------->
As an illustrative example, consider the first causal question of the interest in this exercise, the effect of LDL-c in mid-life on Alzheimer's disease. Here, studies of lipid levels in this time period would require minimal adjustment, whereas studies examining statin use (indirect exposure/intervention) in late-life (indirect population) would be adjusted and down-weighted due to reduced relevance to the causal question.
Once the extent and direction of indirectness has been assessed, total proportional indirectness is then calculated in the same manner as total proportional bias (see Equations \@ref(eq:prop-total-mean) & \@ref(eq:prop-total-var)).
<!----------------------------------------------------------------------->
#### Combining in a bias-/indirectness-adjusted meta-analysis
Using the method reported in Turner _et al._,[@turner2009] once total additive and proportional bias and indirectness are calculated, they are used to define an adjusted estimate for each result included in the synthesis.
The adjusted estimate for result $i$, $\hat{\theta}_i$, is defined as:
<!-- TODO Need to be able to explain the order in which the biases are adjusted for -->
<!-- TODO Talk about validation at some point -->
\begin{equation}
\hat{\theta}_i = \frac{y_i - \mu_{i\beta}^{\mathrm{In}}\mu_{i\delta}^{\mathrm{In}} - \mu_{i\delta}^{\mathrm{B}}}{\mu_{i\beta}^{\mathrm{B}}\mu_{i\beta}^{\mathrm{In}}}
(\#eq:adjusted-mean)
\end{equation}
where $\mu_{i\delta}^{\mathrm{B}}$ and $\mu_{i\beta}^{\mathrm{B}}$ refer to the total additive and proportional bias, and $\mu_{i\delta}^{\mathrm{In}}$ and $\mu_{i\beta}^{\mathrm{In}}$ refer to the total additive and proportional indirectness, respectively. Note that in this exercise, the total additive indirectness $\mu_{i\delta}^{\mathrm{In}} = 1$ because all indirectness was defined _a priori_ as being proportional.
The standard error of this estimate ($\mathrm{SE}(\hat{\theta})$) is then calculated as:
\begin{equation}
\mathrm{SE}(\hat{\theta})=\left(\frac{1}{\mu_{i \beta}^{\mathrm{B}} \mu_{i \beta}^{\mathrm{In}}}\right)^{2}\left(s_{i}^{2}+\left(\mu_{i \beta}^{\mathrm{B}} 2+\sigma_{i \beta}^{\mathrm{B}}\right)\left(\hat{\theta}^{2} \sigma_{i \beta}^{\mathrm{In} 2}+\sigma_{i \delta}^{\mathrm{In} 2}\right)+\sigma_{i \beta}^{\mathrm{B}}\left(\hat{\theta} \mu_{i \beta}^{\mathrm{In}}+\mu_{i \delta}^{\mathrm{In}}\right)^{2}+\sigma_{i \delta}^{\mathrm{B} 2}\right)
(\#eq:adjusted-se)
\end{equation}
Once each individual result had been adjusted using this approach, I performed a random effects meta-analysis using the unadjusted and adjusted results. I also compared the adjusted results under the two scenarios of additive bias. I extracted the overall effect estimates, along with measures of heterogeneity in each case ($\tau^2$, $I^2$).[@higgins2003; @higgins2008]
<!-- TODO Should a fixed or random effects be used, because in theory you are adjusting for all potential causes of heterogeneity. -->
<!-- TODO Sensitivity analysis using only heterogeneity - no change to point estimate - places less importance on the accurate determination of the direction of bias -->
<!----------------------------------------------------------------------->
### Results {#quant-tri-results}
```{r ldlAdBIAMA, include = F}
```
```{r tgVadBIAMA, include = F}
```
#### Single study
As an illustrative example of the process using a single result, the bias and indirectness inherent to the estimate of the effect of statin use on Alzheimer's disease in the CPRD (as analysed in Chapter \@ref(cprd-analysis-heading)) are considered here. For this result, there were three domains at greater than low risk of bias: bias due to confounding, bias due to definition of the outcome, and bias in the selection of the reported result (Figure \@ref(fig:biasDirectionSingle)). In Domain 1 (bias due to confounding), the study did not adjust for _Apo_$\mathcal{E}$ status, which is predicted to mask any true association between statins and Alzheimer's disease (see Section \@ref(cprd-limitations) for a fuller discussion). As such, the bias is predicted to _favour the comparator_ and so the absolute direction of bias is to the right. In Domain 6 (bias in measurement of the outcome), there was a high risk of differential misclassification on the basis of the exposure. Here, a history of statin use may make a diagnosis of vascular dementia more likely than Alzheimer's disease, and so for this outcome, this bias is predicted to _favour the experimental_ (in this case, an intervention). As such, the absolute direction of bias is to the left. In Domain 7 (bias in selection of the reported result), for consistency with the other NRSI, in the absence of a protocol the bias is assumed to be _away from the null_. Given the position of the effect estimate below the null, the absolute direction of bias in this domain is to the left.
<!-- REVIEW YBS - I may have got confused but when you say favour the intervention I am assuming you mean a “protective effect” e.g., OR <1. But if there is differential misclassification so that statin use increases diagnosis of vascular dementia that would show a deleterious effect under the null hence shift to right e.g., OR >1. Perhaps I have misunderstood favour as < risk when in fact you mean > risk. Can you clarify?
This is true for the VaD outcome, but in this case we are interested in the Alzheimer's diease outcome, which would result in under-reporting of AD when on statins and therefore favour the intervention.-->
<!----------------------------------------------------------------------->
(ref:biasDirectionSingle-cap) __Bias direction plot for a single study__ - Summary of result and the predicted direction of bias in each domain of bias for the effect of statin use on Alzheimer's dementia, as estimated in Chapter \@ref(cprd-analysis-heading). Additive biases are characterised using arrows, while the proportional biases are denoted using $<$ and $>$. A question mark indicates that the direction of bias was unpredictable.
(ref:biasDirectionSingle-scap) Bias direction plot example for a single study
```{r biasDirectionSingle, echo = FALSE, results="asis", fig.pos = "H", fig.cap='(ref:biasDirectionSingle-cap)', out.width='100%', fig.scap='(ref:biasDirectionSingle-scap)'}
knitr::include_graphics(file.path("figures/tri/midlife_AD_single.png"))
```
<!----------------------------------------------------------------------->
Indirectness was intentionally not displayed in the plot above because these figures are created using `robvis` which is focused on creating visualisations of risk-of-bias data. In addition, I found it more useful to articulate the indirectness of a result in tabular format. With respect to this example result, I judged it to be at low risk in the population and outcome domains (Table \@ref(tab:singleIndirect-table)). However, the result refers to the effect of statins on Alzheimer's disease risk, rather than LDL-c levels directly, putting it at high risk of indirectness in the intervention/exposure domain.
Overall, assessing the predicted direction of indirectness was more challenging than assessing the predicted direction of bias. In this case, treatment with statins may result in temporary lowering of lipid levels which could thus underestimate the true effect of LDL-c lowering (indirectness towards the null). In contrast, statins may have an effect on Alzheimer's disease through a mechanism other than their LDL-c lowering effect and so the effect may be overestimated (indirectness away from null). As such, I judged the direction of indirectness in this domain to be unpredictable.
<!-- TODO Consider timing of outcome as an issue - but is this a relevance or bias issue -->
<!-- TODO Email Jonathan about how ROBINS-I deals with reverse triangulation? -->
<!----------------------------------------------------------------------->
(ref:singleIndirect-caption) __Assessment of indirectness for a single example study__ - The target setting for the causal question of interest (effect of midlife LDL-c on Alzheimer's disease) and the idealised version of a single example study are compared. The risk and predicted direction of indirectness resulting from discrepancies between the target setting and idealised study are shown.
(ref:singleIndirect-scaption) Assessment of indirectness for a single example study
```{r singleIndirect-table, message=FALSE, results="asis", echo = FALSE}
```
<!----------------------------------------------------------------------->
The unadjusted estimate of the effect of LDL-c on Alzheimer's disease was `r single_unadjusted `. After accounting for the biases and indirectness described, the adjusted result was `r single_adjusted `, representing a minimal shift in the position of the effect estimate but a substantial decrease in precision.
<!----------------------------------------------------------------------->
#### Case study #1: effect of LDL-c at midlife on Alzheimer's disease
From the data sources identified in Section \@ref(tri-data-sources), I identified 21 results relevant to my first causal question of interest. Most (N = 20) were identified via the systematic review,`r ldl_ad_citations` while Chapter \@ref(cprd-analysis-heading) provided an additional estimate via the CPRD analysis. The extent, type and direction of predicted bias for each result can be seen in the bias-direction plot presented in Figure \@ref(fig:ldlAdBiasDirection), stratified by study design.
<!----------------------------------------------------------------------->
(ref:ldlAdBiasDirection-cap) __Bias direction plot for LDL-c/Alzheimer's disease__ - Summary of the unadjusted results and internal biases related to effect of LDL-c on Alzheimer's disease, stratified by study design.
(ref:ldlAdBiasDirection-scap) Bias direction plot for LDL-c/Alzheimer's disease
```{r ldlAdBiasDirection, echo = FALSE, results="asis", fig.pos = "H", fig.cap='(ref:ldlAdBiasDirection-cap)', out.width='100%', fig.scap='(ref:ldlAdBiasDirection-scap)'}
knitr::include_graphics(file.path("figures/tri/midlife_AD.png"))
```
<!-- TODO Bias direction plot is not producing subgroup test for some reason -->
<!----------------------------------------------------------------------->
The results of the unadjusted and the bias-/indirectness-adjusted (under Scenario 1) are shown in Figure \@ref(fig:fpLdlAd). Following adjustment for bias and indirectness, the observed effect attenuates to the null (unadjusted `r unadj_effect`; adjusted `r adj_effect_scenario1`). Additionally, adjustment for bias and indirectness substantially reduced the heterogeneity between studies (unadjusted $\tau^2$ = `r unadj_tau2`, $I^2$ = `r unadj_I2`%; adjusted $\tau^2$ = `r adj_tau2`, $I^2$ = `r adj_I2`%). However, as can been seen from the right panel in Figure \@ref(fig:fpLdlAd), this reduction in heterogeneity between studies is largely achieved via a reduction in the precision of each individual result.
<!-- TODO How effect are fibrates at lowering triglycerides? is it a direct effect, or are there alternative -->
<!----------------------------------------------------------------------->
<!-- TODO This should really be stratified by type -->
<!-- TODO This needs to have the heterogeneity values too -->
(ref:fpLdlAd-cap) __Bias-/indirectness-adjusted meta-analysis of midlife LDL-c on Alzheimer's disease__ - Random-effects meta-analysis of the association of midlife LDL-c with Alzheimer's disease, using unadjusted and bias-/indirectness-adjusted results. Note the substantial decrease in the weight assigned to the result of the CPRD analysis between the two plots, which is due to the high levels of bias/indirectness in this result as described in the above sections.
(ref:fpLdlAd-scap) Bias-/indirectness-adjusted meta-analysis of midlife LDL-c on Alzheimer's disease
```{r fpLdlAd, echo = FALSE, results="asis", fig.pos = "H", fig.cap='(ref:fpLdlAd-cap)', out.width='100%', fig.scap='(ref:fpLdlAd-scap)'}
knitr::include_graphics(file.path("figures/tri/fp_paired_midlife_ldl_ad.png"))
```
<!----------------------------------------------------------------------->
No substantial differences were observed between the summary estimates obtained under the two scenarios of additive bias (Appendix \@ref(appendix-tri-analysis)).
<!-- TODO Need to present overall summary estimate under two scenarios of bias and then send people to the Appendix. -->
<!-- TODO There is a very slight effect in the unadjusted here, but reduces to blah blah on adjustmet -->
<!----------------------------------------------------------------------->
#### Case study #2: effect of triglycerides at midlife on vascular dementia
From the data sources identified in Section \@ref(tri-data-sources), I identified 7 results relevant to the causal effect of triglycerides at midlife on vascular dementia. Four were identified via the systematic review,`r tg_vad_citations` while Chapter \@ref(cprd-analysis-heading) provided two additional estimates from unanalysed datasets (CaPS and Whitehall II), and Chapter \@ref(ipd-heading) provides an estimate for the effect of fibrates, a treatment for hypertriglyceridemia. The extent, type and direction of predicted bias for each result can be seen in the bias-direction plot presented in Figure \@ref(fig:tgVadBiasDirection), stratified by study design.
<!----------------------------------------------------------------------------->
(ref:tgVadBiasDirection-cap) __Bias direction plot for triglycerides/vascular dementia__ - Summary of the unadjusted results and internal biases related to effect of triglycerides on vascular dementia, stratified by study design.
(ref:tgVadBiasDirection-scap) Bias direction plot for triglycerides/vascular dementia
```{r tgVadBiasDirection, echo = FALSE, results="asis", fig.pos = "H", fig.cap='(ref:tgVadBiasDirection-cap)', out.width='100%', fig.scap='(ref:tgVadBiasDirection-scap)'}
knitr::include_graphics(file.path("figures/tri/midlife_VaD.png"))
```
<!----------------------------------------------------------------------------->
Comparison of the unadjusted and bias-/indirectness-adjusted results for the effect of triglycerides on vascular dementia did not demonstrate a substantial difference (unadjusted `r unadj_effect_vad`; adjusted `r adj_effect_scenario1_vad`; Figure \@ref(fig:fpTgVad)), though again, heterogeneity between the adjusted results was greatly reduced (unadjusted $\tau^2$ = `r unadj_vad_tau2`, $I^2$ = `r unadj_vad_I2`%; adjusted $\tau^2$ = `r adj_vad_tau2`, $I^2$ = `r adj_vad_I2`%). Similarly there was minimal difference between the results under the two scenarios of bias (Figure \@ref(fig:fpTgVaDComparison) in Appendix \@ref(appendix-tri-analysis)).
<!-- TODO Need to present overall summary estimate under two scenarios of bias and then send people to the Appendix. -->
<!----------------------------------------------------------------------------->
(ref:fpTgVad-cap) __Bias-/indirectness-adjusted meta-analysis of midlife triglycerides on vascular dementia__ - Random effects meta-analysis of the association of midlife triglycerides and vascular dementia, using unadjusted and bias-/indirectness-adjusted (Scenario 1) results.
(ref:fpTgVad-scap) Bias-/indirectness-adjusted meta-analysis of midlife triglycerides on vascular dementia
```{r fpTgVad, echo = FALSE, results="asis", fig.pos = "H", fig.cap='(ref:fpTgVad-cap)', out.width='100%', fig.scap='(ref:fpTgVad-scap)'}
knitr::include_graphics(file.path("figures/tri/fp_paired_midlife_tg_vad.png"))
```
<!----------------------------------------------------------------------->
## Discussion
### Summary of findings
This chapter has narratively synthesised the evidence identified and produced by the previous chapters, highlighting the absence of any consistent association between various blood lipid fractions and any dementia outcome.
In addition, it has developed a generalised framework for quantitative triangulation, building on existing methods for systematic domain-based risk of bias assessment and bias-/indirectness-adjusted meta-analysis. To illustrate the method, I considered two causal questions: the effect of LDL-c at mid-life with Alzheimer's disease and the effect of triglycerides on vascular dementia. In the quantitative triangulation framework, there was weak evidence for an effect in relation to either of the causal questions considered. The heterogeneity between results produced by different study designs contributing to the analysis was substantially reduced using this method, though this was largely due to a reduction in the precision of each individual result.
<!----------------------------------------------------------------------->
### Limitations
<!-- TODO Data-driven hypothesis testing - only choosing those the two strata that have some suggestion of an effect -->
The quantitative analysis presented in this chapter is subject to some strong methodological limitations, which are discussed in detail below.
<!----------------------------------------------------------------------->
#### Defining reasonable prior distributions for bias and indirectness {#tri-prior-def}
The key limitation of the quantitative synthesis presented in this chapter is the lack of empirical evidence available for the prior distributions of bias and indirectness. While I attempted to address this by informing the prior distributions using data from a previous expert elicitation exercise, the extent of bias/indirectness in that analysis may not generalise to the one presented here.
Ideally, these prior distributions would be based on empirical data on the effect of different domains of bias/indirectness on a result. Most meta-epidemiological studies examining the effect of different methodological issues studied only randomised controlled trials,[@amer2021; @page2016] and the estimates of bias they produce may not hold for non-randomised study designs. However, the prior distributions defined here are broadly comparable to previously reported estimates of the impact of bias in non-randomised trials. For example, a previous simulation study estimated that, in the absence of a true effect of LDL-c, _Apo_$\mathcal{E}4$ would induce a risk ratio of 1.09 with risk of dementia per 1-SD increase in LDL-c.[@iwagami2021] This is comparable to the mean assigned to the prior log-normal distribution mapped to "Moderate" additive bias ($N(0.09,0.05)$, Table \@ref(tab:priorsAdd-table)).
<!-- TODO Also mention here that when the number of results is large, performing elictation for each separately in addition to the risk of bias assessments is going to be difficult. -->
A further limitation is the assumption that the distributions of bias are identical across domains of bias. Previous evidence from meta-epidemiological studies of randomised controlled trials that investigated the effects of different biases indicates that this is unlikely to be the case.[@savovic2018] The generalised framework for quantitative triangulation, presented here and available via the `triangulate` package (see Section \@ref(tri-software)), allows for domain-specific distributions of bias/indirectness. As more information on the effect of specific sources of bias becomes available, such domain-specific distributions should be used.
Finally, for the "bias due to confounding" domain, it may in fact be more reasonable to adjust on a per-confounder basis rather than mapping to a "Moderate" or "High" extent of bias. In this approach, prior distributions are assigned to each of the confounders pre-specified as part of the ROBINS-I/ROBINS-E tools.[@sterne2016] Then, rather than assigning a domain level distribution based on an arbitrary judgement of how many important confounders are missing, the result is adjusted for each confounder not accounted for in the original analysis.
<!----------------------------------------------------------------------->
#### Accuracy of bias-/indirectness-assessment {#tri-accuracy}
A key issue in this analysis is the accuracy of the assessment of bias and indirectness in each result. It has been widely demonstrated that the inter-rater agreement when using risk-of-bias tools is low with regards to the extent of bias assigned to each domain[@hartling2011; @minozzi2019; @minozzi2020] Though not previously studied, given the difficulty in assessing the predicted direction of bias, agreement on this aspect of the tools is likely to be lower still.
A further issue in this regard is that the rigour of the tools may not be equivalent. This issue can be illustrated by the Ostergaard _et al._ study[@ostergaard2015] having low risk of bias across all domains and thus receiving a substantial proportion of the weight in the first case study (Figure \@ref(fig:fpLdlAd)). While the RoB suite of tools (RoB 2.0, ROBINS-I, ROBINS-E) were developed by an expert working group and have been widely used, the Mamluk _et al._ tool[@mamluk2020] used to assess risk of bias in Mendelian randomisation studies was designed by the authors for use in their own review. If the Mamluk tool is failing to adequately assess bias in Mendelian randomisation studies, then any analysis based on its assessments is likely to be subject to bias. This observation is particularly problematic given the default position of the ROBINS-I and ROBINS-E tool to require a "Moderate" judgement in the confounding domain. Under these conditions, it is possible that a well-performed cohort study (low risk of bias across everything other that bias due to confounding) will be downweighted versus a potentially poor MR study.
<!----------------------------------------------------------------------->
#### Low and critical risk of bias
As illustrated in Table \@ref(tab:robLevelsMapping-table), studies at critical risk of bias were not included in the analysis. While this is in line with best practice, there is theoretically no reason why studies with this extent of bias could not be included in the proposed framework. However, the estimation of an appropriate prior distribution for the effect of "Critical" bias would be substantially more challenging, and so was avoided here.
A related issue is the "Low" risk of bias judgement. In this analysis, I assumed that domains at "Low" risk of bias did not require any adjustment. However, "Low" risk of bias is not equivalent to the absence of bias, and potentially should still be adjusted for, if only minimally. In this case, defining the predicted direction of bias would be particularly challenging in the absence of obvious sources of bias.
<!----------------------------------------------------------------------->
#### Combination of different effect measures
A final limitation of this analysis is the synthesis of different effect estimates. As discussed in Section \@ref(sys-rev-analysis-overview), hazard ratios are not directly comparable to odds ratios because they inherently account for person time-at-risk.[@mckenzie2019] Non-randomised studies (NRSI/NRSI) of dementia outcomes reported hazard ratios and Mendelian randomisation studies reported odds ratios, and as such the resulting synthesis may be biased by the integration of these two different effect measures. This is primarily a concern for common outcomes because the odds, risk and hazard ratios approximate each other when the outcome is rare.
In the case of common outcomes, an extra step may be needed in the framework to assesses whether it is reasonable to synthesis across different effect estimates.
<!-- REVIEW Could add that ORs could be converted to RRs using a measure of "hyptothecial" risk -->
<!-- TODO An extra step of the process should be the consideration of whether it is reasonable to synthesis across different effect estimates. Fine for rare, not for common. Fancy models that can be used to synthesis effect estimates are still compatabile with this approach as each invidual estimate is first adjusted and the adjusted estimate can then be used with whatever meta-anlaysis technique you like. -->
<!----------------------------------------------------------------------->
### Strengths
<!-- TODO An additional strength could be the fact that I explicitly sought to include both direct and indirect evidence -->
The core strength of this analysis is that it is based on a comprehensive systematic review, supplemented by two additional primary studies to fill in the identified evidence gaps. In general, triangulation of any sort should be considered a natural extension of evidence synthesis, and therefore should follow best practice in relation to finding and critically assessing all relevant information. Additionally, the approach presented here also builds on recent developments in bias assessment to "explode out" the component results of a meta-analysis and consider the effects of bias/indirectness in each separately.
A further advantage of this method comes from the ability to specify the prior distributions for each level of bias/indirectness in advance of performing the assessments. Expert elicitation of the extent of bias/indirectness using a panel of methodologists and topic experts is a powerful technique but the point at which it is deployed in the framework is important. In previous attempts at bias-/indirectness-adjusted meta-analysis, the extent of bias in each study was assessed via an elicitation process,[@thompson2011; @turner2009; @wilks2011] during which experts were aware of the results of each analysis. This approach is potentially subject to differential misclassification of the impact of bias/indirectness on the basis of the results because there is no way to ensure that results at a similar level of bias for confounding (for example) are being adjusted by the same amount across studies. As an illustrative example, if experts are influenced by the knowledge of the effect estimate in a study, then a result at "Moderate" risk of confounding and demonstrating a stronger protective effect may receive greater adjustment than a study at the same risk of confounding but with a more modest effect. This is likely to be particularly problematic where experts have preconceived ideas about the true effectiveness of the intervention.
Separating the assessment of bias/indirectness from the assignment of modifying values to each judgement will limit the potential for this misclassification. Using the framework proposed, reasonable modifying distributions for each level of bias can be defined _a priori_ by the study team using the elicitation procedure detailed previously.[@turner2009] This approach would be similar to how important confounders and co-interventions are defined in advance when performing ROBINS-I/ROBINS-E assessments.[@sterne2016] As an example, expert elicitation could be used to _a priori_ define an additive bias distribution of N(0.1,0.7) for "Moderate" risk of bias in Domain 1 of the ROBINS-I tool (bias due to confounding). Risk-of-bias assessments would then be performed, and each result at "Moderate" risk of bias in Domain 1 is adjusted using this pre-specified distribution.
Finally, accounting for bias using an incorrect prior distribution is no less problematic than synthesising and drawing conclusions from effect estimates that are treated as unbiased, a common occurrence in systematic reviews.[@katikireddi2015] While this analysis may be limited by the absence of empirically-derived adjustment distributions, it at the very least acknowledges the uncertainty of evidence due to bias/indirectness via a reduction in precision.
<!-- REVIEW YBS I was struck by how much the 95% Cis were widened with CPRD. Is there a risk we are adding to much uncertainty and therefore down weighting this too much? -->
<!----------------------------------------------------------------------->
### Future research
<!-- REVIEW Good to mention direct adjustment for bias, e.g., Greenlands writings -->
An obvious avenue for future work in relation to quantitative triangulation is the identification of empirical prior distributions for the effect of bias/indirectness in non-randomised studies using meta-epidemiological approaches. This will be substantially more challenging than examining the effect of bias in RCTs,[@savovic2018] given the absence of an underlying database of meta-analysis of non-randomised studies.
Future development of this framework should also account for bias at the analysis level in terms of missing evidence.[@zotero-15123] For example, there is at least one known study relevant to the triglycerides/vascular dementia case study that reported a non-significant result but provided insufficient details to be included in the analysis.[@chiang2007] Ideally, a quantitative triangulation framework would further account for this proportional meta-bias (seeing as bias due to missing evidence is most likely to bias away from the null due to publication bias mechanisms), in addition to result-specific biases.
Appreciation that both the risk-of-bias and indirectness of a result should be examined is growing. Indeed, some existing tools for the assessment of diagnostic test accuracy[@whiting2011] and prediction models[@moons2019] consider indirectness (termed applicability in these tools) alongside the assessment of bias. Future iterations of the core risk-of-bias tools (RoB 2, ROBINS-I, ROBINS-E) could take this into account, though there is a competing argument that indirectness is already handled via tools such as the GRADE framework.[@guyatt2011] In addition, simple steps like harmonising the risk-of-bias judgements across tools and making the direction of bias question mandatory (even if users default to "Unpredictable") would represent an improvement in the tools and encourage users to think about how the biases assessed affect the corresponding result. Similarly, software that implements a risk-of-bias tool should allow for direction/extent of bias question and carefully consider how this data will be exported.
<!----------------------------------------------------------------------->
### Outputs {#tri-software}
There are two key outputs from this chapter. The first is an R package, `triangulate`, available at [https://github.com/mcguinlu/triangulate](https://github.com/mcguinlu/triangulate). Personal communication with the authors of the bias-/indirectness-adjusted meta-analysis approach resulted in the STATA code to implement the adjusted model.[@turner2009] This has since been generalised as part of the `triangulate` R package to enable other users to apply the approach detailed here. The package allows for preprocessing of domain-based risk-of-bias data to correctly identify the direction of bias relative to the effect estimate, use of domain-specific prior distributions, production of bias direction plots (via the tool described in Appendix \@ref(appendix-robvis)), calculation of bias/indirectness-adjusted estimates for each result and synthesis of these adjusted results in a standard random-effects meta-analysis.
<!-- REVIEW Describe what you contributed here a little better --->
The second key output is an example dataset for the LDL-c/Alzheimer's disease causal question addressed in this chapter. Available via the `triangulate` package, this data will provide developers of new triangulation methods with an example dataset on which to test new tools, something which seriously hindered development of this work.
<!----------------------------------------------------------------------->
## Summary
* In this chapter, I narratively described the evidence identified or produced in previous chapters and highlighted the issues with a qualitative approach to triangulation when there are several sources of evidence.
* I then illustrated a proposed generalised framework for quantitative triangulation, exploiting existing methods in evidence synthesis and recent developments in the assessment of different risks of bias. I illustrated this method using two case studies: the effect of mid-life LDL-c on Alzheimer's disease, and the effect of mid-life triglycerides on vascular dementia.
* I discussed the limitations of this proposed method, given the absence of empirically based priors for the impact of bias/indirectness, and proposed future research (i.e. meta-epidemiological studies) that would address them.
* In summary, quantitative triangulation is a promising field that should be considered a natural extension of evidence synthesis. The implications of the synthesised evidence presented here to clinical practice, public health and future research is considered in the following chapter.