Case Study: Building a Reliable Analytics Framework for Hybrid Instruction Research

---

Context for this hybrid instruction analytics case study

A PhD dissertation examined hybrid instruction analytics by combining online modules with in‐person classes. Data came from multiple sources—quiz scores, engagement logs, survey responses—collected across four courses over two semesters. The research team needed a scalable, reproducible analytics framework to manage evolving datasets and automate reporting for diverse stakeholders.

---

1. Data Version Drift

• Challenge: Team members maintained separate Excel files, resulting in inconsistent versions of the same data.
• Solution:
  1. Central Masterchart: All raw and cleaned data merged into a single masterchart.csv.
  2. Script-Driven Transforms: R scripts and Excel VBA macros read and write only to the masterchart, eliminating manual copy-paste steps.
  3. Version Logs: Each transformation step appends a timestamp and Git commit hash to a changelog table for auditability.

---

2. Manual Dashboard Workload

• Challenge: Creating hundreds of charts by hand in Excel led to formatting errors and inconsistent naming.
• Solution:
  1. VBA Automation: A CreateDashboard() macro loops through predefined variable lists, inserts pivot tables, and applies uniform chart templates.
  2. R Script Alternative: generate_dashboards.R uses ggplot2 to recreate Excel charts in PDF, ensuring identical color palettes and axis scales.
  3. Naming Conventions: Automated file names include date, variable name, and cohort, e.g. dashboard_Q1_2025-06-20_child.pdf.

---

3. Assumption Oversight for hybrid instruction analytics

• Challenge: Early analyses ran t‑tests and ANOVAs without checking normality or variance equality.
• Solution:
  1. Automated Diagnostics: A helper function runs Shapiro–Wilk and Levene’s tests before each parametric analysis.
  2. Branch Logic: If p < 0.05 (assumption violation), the pipeline switches to Wilcoxon or Kruskal–Wallis tests automatically.
  3. Report Flags: The final report highlights any assumption failures and shows both parametric and nonparametric results for transparency.

---

4. Opaque Reporting for Stakeholders

• Challenge: Faculty and administrators found raw tables of coefficients and p‑values overwhelming.
• Solution:
  1. Top‑Line Summaries: A one‑page dashboard document opens with plain‑language statements like “68 % of students preferred online content.”
  2. Annotated Charts: Callout labels highlight significant group differences directly on bar and line graphs.
  3. Interactive Excel Workbooks: Dropdown filters let stakeholders explore results by course, week, or demographic without touching the code.

---

5. Pipeline Reproducibility

• Challenge: Hard‑coded file paths and manual steps made it impossible for others to rerun analyses.
• Solution:
  1. Parameterization: All paths and settings moved to a single YAML config file (config.yml).
  2. Enforced Directory Structure: Scripts assume a standard folder tree—/data/raw, /data/processed, /results, /reports.
  3. Continuous Integration: GitHub Actions automatically runs the full pipeline on each push, verifying that masterchart.csv transforms and report generation complete without errors.

---

Best Practices Checklist from hybrid instruction analytics case study

• Masterchart Governance:
  • Use one CSV as the single source of truth.
  • Validate schema with assertion tests.
  • Log macro and script versions.
• Transparent Transformations:
  • Comment each code block.
  • Produce and save pre‑/post‑transformation density plots.
• Scenario Logging:
  • Store “What If” scripts in a separate folder.
  • Auto‑generate comparative summary tables for each scenario.
• Modular Code & Peer Review:
  • Split R and VBA into discrete functions.
  • Develop in Git feature branches.
  • Require pull requests and peer review before merging.
• Dual Reporting:
  • Provide an executive summary with key findings.
  • Include a full technical appendix in PDF form.
  • Supply interactive Excel dashboards for non‑technical audiences.
• Audit‑Ready Deliverables:
  • Create a final Data Audit PDF detailing every transformation step, assumption check, and sign‑off from stakeholders.

---

Outcome:
This robust analytics framework—built on a centralized masterchart, automated dashboards, assumption checks, and CI pipelines—ensured the hybrid instruction research was dependable, transparent, and reproducible from raw data to stakeholder reports.

---

Want to explore more PhD-level case studies? Check out our Comprehensive Case Studies on PhD Statistical Analysis guide page.