1 Overall Rationale: Why Reproducibility Matters in this date

Reproducibility in R has derived from theory to practice. After handling numerous data science projects, I have learned that reproducibility is rather fundamentally about maintainability and the ability to return to a project months or years later and have it actually work, not just following best practices. Thus, it matters.

The core problem we face as R practitioners is simple: code that runs perfectly today may fail to other machines or in future executions. Any programming languages you’ll use, not just R, may encounter:

1. Packages update
2. Dependencies changes
3. Evolving versions evolve
4. System libraries get upgraded.

What worked flawlessly on your machine might throw cryptic errors on a colleague’s computer. A pipeline that ran successfully six months ago might inexplicably break when you try to reproduce your results.

This post outlines my known reproducibility tools in current 2026, built around five core tools: renv for package management, Git for version control, targets for workflow orchestration, box for code organization, and rixpress (powered by rix) for full-stack reproducibility when standard approaches are insufficient.

I am gonna try to explain what these tools do, and when and why you should use each one, building up from simple projects to complex, long-lived data pipelines.

Just remember: Over-engineering a simple analysis with unnecessary infrastructure wastes time and creates maintenance burden. Conversely, under-engineering a complex project leads to fragility and technical debt.

2 Option 1: {renv} + git + {pak}

The renv package got more and more stable after years of development. It is nice that they are listening to people’s mindful criticism about its flaws, and this is the best thing we have when dealing with freezing the working environment. It is so stable, it sync with Git version properly, not perfect but that’s the best thing we had for now. Also, my recommendation is that pair it with pak as it resolves the package you want to install fast, and also it syncs well with the initialized renv project.

renv::restore()

.Rproj.user
.Rhistory
.RData
.Ruserdata
renv/library/
renv/local/
renv/staging/
*.csv
*.rds
data/raw/*
.DS_Store
*.Rproj

3 Option 2: {targets}

Although I admit that I don’t use targets that often, this package is part of my stack. Take note that in this part, I am only gonna explain the surface part of this tooling.

# _targets.R
library(targets)
library(tarchetypes)

tar_option_set(packages = c("dplyr", "readr", "ggplot2"))

list(
  tar_target(raw_data_file, "data/raw.csv", format = "file"),
  tar_target(raw_data, read_csv(raw_data_file)),
  tar_target(clean_data, raw_data |> filter(!is.na(value))),
  tar_target(model, lm(outcome ~ predictor, data = clean_data)),
  tar_target(plot, {
    ggplot(clean_data, aes(predictor, outcome)) +
      geom_point() +
      geom_smooth(method = "lm")
  })
)

targets::tar_make()

4 Option 3: {box}

I’ve been using this package lately for my R projects and blogs, just to import the namespace of the package I needed.

Hot take: I freaking hate using source(), I feel like I bit my tongue off whenever I use it. I abandoned the source() habit for (quite not) a long time now after discovering this package.

# I hate doing this:
source("R/data_processing.r")
source("R/plotting.r")

# Instead, do this:
box::use(
    dm = ./data_munging[clean_data, validate_data],
    ./plotting[make_plot], 
    ./plotting
)

5 Option 4: {rix} and {rixpress}

# Install rix
# install.packages("rix")
pak::pkg_install("rix")

box::use(rix[rix])

rix(
    r_ver = "4.4.1",
    date = "2025-11-30",
    packages = c("dplyr", "ggplot2", "tidyr", "sf", "box"),
    system_packages = c("gdal", "proj"),
    project_path = ".",
    overwrite = TRUE,
    ide = "rstudio"
)

nix develop

# Install rixpress
# remotes::install_github("b-rodrigues/rixpress")
# devtools::install_github("b-rodrigues/rixpress")
pak::pak("b-rodrigues/rixpress")

box::use(
    rixpress[
        rxp_pipeline, rxp_r, rxp_py, rxp_populate
    ],
    dplyr[mutate, keep_when = filter], 
    readr[read_csv, write_csv]
)

pipeline = list(
    # R step: load and clean data
    rxp_r(
        name = "clean_data",
        expr = {
            read_csv("data/raw.csv") |> 
                keep_when(!is.na(value)) |> 
                write_csv("data/clean.csv")
        },
        rix_env = "r_env_4.4.1"
    ),
    
    # Python step: run ML model
    rxp_py(
        name = "train_model",
        expr = "
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
import pickle

data = pd.read_csv('data/clean.csv')
X = data[['feature1', 'feature2']]
y = data['target']

model = RandomForestRegressor()
model.fit(X, y)

with open('model.pkl', 'wb') as f:
    pickle.dump(model, f)
    ",
        deps = "clean_data",
        python_version = "3.11"
    ),
    
    rxp_r(
        name = "report",
        expr = {
            rmarkdown::render("report.Rmd")
        },
        deps = "train_model",
        rix_env = "r_env_4.4.1"
    )
)

rxp_populate(pipeline, build = TRUE)

6 Option 5: Reporting with R Markdown / Quarto

Reproducible reporting is where the rubber meets the road. You can have perfect package management, clean code organization, and efficient workflows. Those options above doesn’t solves actual reporting, and on how you presented the code you just run. This is where R Markdown / Quarto comes in.

title: "Report for February 07, 2026"
author: "John Doe"
output: html_document

format: revealjs
filters:
    - social-share
    - collapse-output
engine: knitr

7 Combining Tools in Practice

Can you combine the options being placed here? Straightforwardly, yes.

1. In my work, I use options 1 and 3. Having them for reproducibility and maintainability, especially for my own work and make my work shareable to others, is more than enough.

2. Options 1 + 2 + 4 is good when you’re building production-grade data pipelines that need to run reliably over years.

3. You can combine either of the first 4 options then option 5 for reporting.

Except for one thing: targets + box, and it is being discussed here why it is a case.

I’ll make it simple (this is at as far as I know):

1. targets relies on static analysis through codetools to discover symbols in your target expressions and build a proper dependency graph (DAG) without running the code. It needs functions and objects to be statically visible.

2. In contrast, box resolves functions at runtime inside module environments (e.g., via $), which hides internal symbols from static analysis. As a result, targets may not fully “see” the real function-level dependencies inside box modules.

8 Special Mention

Here are other tools used for reproducibility, but more niche, superseded, and/or known less (by me):

1. reproducible - it is now on its version 3.0.0, and expect to have some breaking changes. This is niche.
2. packrat - this is now a legacy, mostly superseded by renv.
3. groundhog - purely for packages, specialized in timing. I don’t know much about this one.
4. drake - mostly superseded by targets.
5. reprex - only good for providing reproducible example.

1 Introduction

Pandas 3.0 is the biggest update in the library’s history, bringing breaking changes that fix long-standing confusions and performance issues. If you’re new to pandas, you’re starting with a cleaner API. If you’re migrating, this guide covers what you need to know.

2 The Three Big Changes

Here’s what it potentially breaks your pandas pipeline.

import pandas as pd

ser = pd.Series(["apple", "banana", "cherry"])
print(ser.dtype)  
#>>> str 

# This breaks
if df['column'].dtype == 'object':
    process_strings(df['column'])

# Use this instead
if pd.api.types.is_string_dtype(df['column']):
    process_strings(df['column'])

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'score': [85, 92, 78]
})

high_scorers = df[df['score'] > 90]

# Modify filtered data
high_scorers['passed'] = False

# Doesn't work anymore
df['score'][df['name'] == 'Alice'] = 100

# Use loc instead
df.loc[df['name'] == 'Alice', 'score'] = 100

# Old way
df = df.assign(temp_f = lambda x: x['temp_c'] * 9 / 5 + 32)

# New way with pd.col()
df = df.assign(temp_f = pd.col('temp_c') * 9 / 5 + 32)

# Arithmetic
df = df.assign(
    total=pd.col('price') * pd.col('quantity'),
    discounted=pd.col('total') * 0.9
)

# String operations
df = df.assign(
    name_upper=pd.col('name').str.upper(),
    is_alice=pd.col('name') == 'Alice'
)

# Filtering
expensive = df.loc[pd.col('price') > 100]

df.groupby('category').agg(pd.col('value').mean())

3 Migration Strategy

# Upgrade to 2.3, fix warnings
pip install pandas==2.3.0

# Then upgrade to 3.0
pip install pandas==3.0.0

# Enable new behavior before upgrading
pd.options.mode.copy_on_write = True
pd.options.future.infer_string = True

4 Performance Improvements

With PyArrow-backed strings, expect significant gains:

• String operations: up to 10x faster
• Memory usage: ~50% reduction for string columns
• Reduced unnecessary copies with Copy-on-Write

5 Best Practices

1. Use pd.col() for clarity

df = df.assign(
    full_name = pd.col('first') + ' ' + pd.col('last'),
    is_senior = pd.col('age') >= 65
)

2. Always use .loc for assignment

df.loc[mask, 'column'] = value

3. Trust Copy-on-Write (no defensive .copy())

subset = df[df['value'] > threshold]
subset['processed'] = True  # df unchanged!

4. Use type-checking functions

if pd.api.types.is_string_dtype(df['col']):
    # Handle strings

6 Conclusion

Don’t celebrate yet, although Pandas 3.0 fixes years of confusing behavior with three major changes: proper string types, predictable copy semantics, and cleaner column syntax, there are still some inconsistencies—this library is not the as other data libraries out there, e.g. dplyr, polars.

If you’re new, you’re learning the modern API. If you’re migrating, upgrade to 2.3 first, fix warnings, then jump to 3.0. The changes might feel disruptive, but they solve real problems and make pandas faster and more composable than before.

1 Introduction to PyTorch

I’ve been writing Python to do deep learning stuffs for ~2 years now. I started from tensorflow then later PyTorch and it has become my go-to framework for deep learning, and for good reason. Developed by Meta’s AI Research lab, PyTorch is an ergonomic framework to buildi neural networks in Pythonic approach that feels natural to any Python developer. Unlike some other frameworks that can feel overly abstracted, PyTorch strikes the perfect balance between ease of use and fine control.

No worries, I am still using tensorflow / keras in this date.

2 Why PyTorch?

After working with TensorFlow, Keras, and other frameworks, I consistently find myself reaching for PyTorch. Here’s why:

Dynamic Computation Graphs: PyTorch uses dynamic computational graphs (define-by-run), meaning the graph is built on the fly as operations are executed. This makes debugging significantly easier—you can use standard Python debugging tools and print statements to inspect tensors at any point in your code.

Pythonic: Not gonna lie, using “Pythonic” word makes me feel cringe. Anyways, PyTorch feels like native Python. The API is less abstract but still clean, consistent, and follows Python conventions. If you know NumPy, you’ll feel right at home with PyTorch tensors.

Research-Friendly: PyTorch has always been ideal for research and experimentation, well aside from JAX. You can easily modify architectures, implement custom layers, and try novel ideas without fighting against the framework. Perfect for ML researchers like me.

Strong Community and Ecosystem: PyTorch has excellent documentation, a vibrant community, and a rich ecosystem of libraries like PyTorch Lightning, Hugging Face Transformers, and torchvision.

3 Getting Started

# CPU version
pip install torch torchvision torchaudio

# GPU version (CUDA 11.8)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# CPU version
uv pip install torch torchvision torchaudio

# GPU version (CUDA 11.8)
uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

import torch

4 Building Neural Networks

PyTorch’s nn.Module class provides a clean way to define neural networks:

import torch.nn as nn
import torch.nn.functional as F

class SimpleNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, output_size)
    
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

After defining the module, then you can simply instantiate the model:

model = SimpleNN(input_size=784, hidden_size=128, output_size=10)

5 Training Loop

The training process in PyTorch is explicit and transparent:

1. Define loss and optimizer

   import torch.optim as optim
   
   criterion = nn.CrossEntropyLoss()
   optimizer = optim.Adam(model.parameters(), lr=0.001)

2. Run the training loop:

   for epoch in range(num_epochs):
       for batch_idx, (data, target) in enumerate(train_loader):
   1        optimizer.zero_grad()
   
   2        output = model(data)
           loss = criterion(output, target)
   
   3        loss.backward()
   
   4        optimizer.step()
   
           if batch_idx % 100 == 0:
               print(f'Epoch: {epoch}, Loss: {loss.item():.4f}')
   

   1

   Zero gradients

   2

   Forward pass

   3

   Backward pass

   4

   Update weights

6 PyTorch vs. Other Frameworks

1. vs. TensorFlow/Keras: While TensorFlow 2.x has improved significantly with eager execution, PyTorch still seems much closer to the heart of DL to me. Sure, {keras} module is great for quick prototyping, but I find PyTorch’s explicit approach gives me better control and understanding of what’s happening under the hood.

2. vs. JAX: JAX is actually excellent (been using this for a year now) for numerical computing and offers impressive performance, but PyTorch’s ecosystem, documentation, and community support are currently unmatched, which includes transformers, etc. For production applications and research, I would argue PyTorch provides better tooling and debugging capabilities.

7 Key Features I Love

Autograd: It has Automatic differentiation (autograd) and it is seamless. Just call .backward() on your loss, and PyTorch handles the gradient computation.

DataLoader: Built-in utilities for efficient data loading, batching, and shuffling make preprocessing a breeze.

TorchScript: When you need to deploy models to production, TorchScript allows you to serialize and optimize your models.

Mixed Precision Training: Easy to implement with torch.cuda.amp for faster training and reduced memory usage.

8 Extra: torch implementation in R

PyTorch lives in LibTorch, its C++ distribution—the actual computational core where tensors, autograd, and neural network primitives are implemented. Python is “just” one of the frontends that talks to this core.

Thanks to {Rcpp}, the {torch} package in R is able to interface directly with LibTorch. Rather than wrapping Python or relying on {reticulate}, {torch} binds to PyTorch’s C++ backend itself, exposing the same tensor operations, automatic differentiation engine, and neural network building blocks to R users.

This design choice is crucial. It means that {torch} in R is not a secondary or reimplemented version of PyTorch, but another first-class frontend to the same underlying engine. Training a model in R with {torch} executes the very same C++ code paths as training in Python—only the surface language changes. As a result, performance characteristics, numerical behavior, and model semantics remain consistent across languages, while still allowing R users to work in an idiomatic R style.

9 Conclusion

PyTorch has earned its place as the dominant framework in deep learning research and is increasingly popular in production environments. Its intuitive design, flexibility, and powerful features make it my framework of choice. Whether you’re just starting out or you’re an experienced practitioner, PyTorch provides the tools you need to build sophisticated models without unnecessary complexity.

The learning curve is gentle, the documentation is excellent, and the community is supportive. If you haven’t tried PyTorch yet, I highly recommend giving it a shot—you might find, like I did, that you never want to go back.

1 Introduction

Python dominates the AI/DL space very well (Hugging Face, diffusion models, large-scale RL), and it’s such a pleasure to deploy those model into production at scale. However, many data scientists, statisticians, bioinformaticians, and analysts continue working in R because:

• Data naturally lives in tidy data frames, keeping preprocessing, exploration, and visualization in one language
• tidymodels provides a consistent, production-ready modeling grammar
• torch now offers excellent native GPU support without requiring Python installation
• Strong deployment options exist (Shiny, Posit Connect/Workbench, plumber APIs, vetiver)
• Statistical modeling and deep learning can coexist seamlessly in the same session

2 Why Neural Networks in R?

You see, R typically isn’t the best choice for training 100B-parameter LLMs or real-time video models, but it excels at:

• Tabular deep learning
• Research prototyping where statistics and interpretability matter
• Teaching deep learning in statistics-focused curricula

Before diving into technical details, let’s address a key question: why choose R for deep learning when Python has such a robust ecosystem?

For many R users, the answer is straightforward: workflow integration.

3 The R Neural Networks Ecosystem in 2026

Let’s examine the available tools.

box::use(
    torch[randn = torch_randn, torch_matmul]
)

x = randn(c(5, 3))
y = randn(c(3, 2))
z = torch_matmul(x, y)

box::use(
    keras[
        model_sequential = keras_model_sequential, 
        to_categorical, 
        layer_dense, 
        optimizer_adam, 
        compile, fit, 
        evaluate
    ],
    stats[predict]
)

x = as.matrix(iris[, 1:4])
y = {as.integer(iris$Species) - 1} |> to_categorical(num_classes = 3)

model = 
    model_sequential() |>
    layer_dense(
        units = 16,
        activation = "relu",
        input_shape = ncol(x)
    ) |>
    layer_dense(
        units = 16,
        activation = "relu"
    ) |>
    layer_dense(
        units = 3,
        activation = "softmax"
    )

model |>
    compile(
        optimizer = optimizer_adam(learning_rate = 0.01),
        loss = "categorical_crossentropy",
        metrics = "accuracy"
    )

history =
    model |>
    fit(
        x,
        y,
        epochs = 50,
        batch_size = 16,
        validation_split = 0.2,
        verbose = 1
    )

model |>
    evaluate(x, y)

pred = predict(model, x)

class_pred = max.col(pred) - 1

4 High-Level Interfaces

This is where things get interesting. Raw torch requires substantial boilerplate—defining modules, training loops, data loaders, etc. Several packages have emerged to simplify this workflow:

• luz: A high-level interface for torch with built-in training loops
• tabnet: Implements TabNet architecture specifically for tabular data
• brulee: High-level modeling functions with torch
• mlr3torch: Bridges torch deep learning into the mlr3 ecosystem
• kindling: (My package!) Simplified torch neural network modeling with reduced boilerplate

5 Introducing {kindling}: Torch with Less Typing

Here’s the problem I aimed to solve: torch is powerful for deep learning and scientific computing, but verbose and less expressive (no offense), while tidymodels has an ergonomic API, thanks to tidyverse for traditional ML, but has limited deep learning stacks and integrations.

The kindling package goes further beyond by enabling you to:

1. Customize the number of hidden layers
2. Specify activation functions for each layer, and their specific parameters (e.g. lambd from nnf_softshrink() function)
3. Tune neural network architecture while incorporating both capabilities above

I built this package for convenience, consistency, and ease of modeling. In its current implementation, it’s limited to data frames only, and planning to extend this for image datasets.

6 References

Falbel D, Luraschi J (2023). torch: Tensors and Neural Networks with ‘GPU’ Acceleration. R package version 0.13.0.

Goodfellow I, Bengio Y, Courville A (2016). Deep Learning. MIT Press.

Kuhn M, Wickham H (2020). Tidymodels: a collection of packages for modeling and machine learning using tidyverse principles.

Wickham H (2019). Advanced R, 2nd edition. Chapman and Hall/CRC.

Keydana S (2023). Deep Learning and Scientific Computing with R torch

1 What do I love about aes()?

The aes() function is a domain-specific language (DSL) embedded inside R — a Lisp-y function that deeply ties with lazy evaluation, quosures, and deep integration with the tidyverse.

What keeps pulling me back is how elegantly it balances simplicity for beginners with incredible power for advanced use — all without ever feeling clunky.

2 Mapped vs Fixed Aesthetics

One of the first things to understand is the distinction between mapped and fixed aesthetics.

box::use(
    ggplot2[ggplot, aes, geom_point]
)

ggplot(mtcars, aes(x = mpg, y = hp, color = factor(cyl))) +
    geom_point()

ggplot(mtcars, aes(x = mpg, y = hp)) +
    geom_point(color = "blue")

When you put an aesthetic inside aes(), it’s mapped to a variable. Outside aes(), it’s fixed to a constant value. This distinction is crucial but often trips up beginners.

3 The group Aesthetic

The group aesthetic controls how ggplot2 groups data for statistical transformations and geoms like lines.

box::use(
    ggplot2[ggplot, aes, geom_line]
)

ggplot(mtcars, aes(x = mpg, y = hp)) +
    geom_line()

ggplot(mtcars, aes(x = mpg, y = hp, group = cyl)) +
    geom_line()

Often, group is set automatically when you use color, linetype, or fill, but sometimes you need explicit control.

4 Stage-Based Transformations: after_stat() and after_scale()

I already made a blog talking about delaying aes() arguments, you can check it here. These functions let you compute aesthetics based on statistical transformations or after scale transformations.

box::use(
    ggplot2[
        ggplot, aes, geom_histogram, geom_point, 
        after_stat, after_scale, alpha
    ]
)

1. Color bars based on computed count from stat_bin

   ggplot(mtcars, aes(x = mpg)) +
       geom_histogram(aes(fill = after_stat(count)), bins = 10)

   

2. Use after_scale() to modify colors after scale mapping

   ggplot(mtcars, aes(x = mpg, y = hp, color = factor(cyl))) +
       geom_point(aes(color = after_scale(alpha(color, 0.5))), size = 4)

   

after_stat() accesses variables computed by statistical transformations (like count, density, etc.). after_scale() works with the final scaled aesthetic values, allowing post-processing of colors, sizes, and more.

5 Tidy Evaluation and Programming with aes()

When you’re writing functions that create ggplot2 plots, you need to programmatically pass variable names. This is where tidy evaluation with { } (curly-curly) and !! (bang-bang) comes in.

1. The double curly brackets { } is more recommended

   plot_scatter = function(data, x_var, y_var, color_var) {
       box::use(
           ggplot2[
               ggplot, aes, geom_point, theme_minimal
           ]
       )
   
       ggplot(data, aes(x = {{ x_var }}, y = {{ y_var }}, color = {{ color_var }})) +
           geom_point() +
           theme_minimal()
   }
   
   plot_scatter(mtcars, mpg, hp, cyl)

   

2. With rlang::ensym(), it accepts both string and a bare column

   plot_scatter_string = function(data, x_var, y_var) {
       box::use(
           ggplot2[ggplot, aes, geom_point],
           rlang[ensym]
       )
   
       ggplot(data, aes(x = !!ensym(x_var), y = !!ensym(y_var))) +
           geom_point()
   }
   
   plot_scatter_string(mtcars, "mpg", "hp")

   

   # plot_scatter_string(mtcars, "mpg", hp)
   # plot_scatter_string(mtcars, mpg, "hp")
   # plot_scatter_string(mtcars, mpg, hp)

This works because aes() uses quosures (quoted expressions with their environment) internally, which are part of rlang’s non-standard evaluation framework.

6 Integration with the Tidyverse

The aes() function plays beautifully with tidyverse workflows because they share the same evaluation model.

box::use(
    dplyr[mutate, case_when], 
    ggplot2[ggplot, aes, geom_point, labs, theme_minimal]
)

mtcars |> 
    mutate(
        efficiency = mpg / hp,
        cyl_category = case_when(
            cyl == 4 ~ "Small",
            cyl == 6 ~ "Medium",
            cyl == 8 ~ "Large"
        )
    ) |> 
    ggplot() +
    aes(x = wt, y = efficiency, color = cyl_category) + 
    geom_point(size = 3) +
    labs(title = "Efficiency by Weight and Engine Size") + 
    theme_minimal()

You can pipe data through dplyr transformations and directly into ggplot2, and all the column names work seamlessly. This should be trivial to you since this is more taught by most ggplot2 tutorials.

7 Other aesthetic ‘weaponries’ you want to use

Beyond the common patterns, aes() has several advanced capabilities that showcase its flexibility.

Here, I can name few:

box::use(
    ggplot2[
        ggplot, aes, geom_point, geom_smooth, labs, 
        geom_line, theme_minimal
    ]
)

ggplot(mtcars, aes(x = wt, y = mpg, color = factor(cyl))) +
    geom_point(size = 3) +
    # Override color for this layer only
    geom_smooth(
        aes(color = NULL), 
        method = "lm", 
        color = "#061E29", 
        se = FALSE,
        linetype = "dashed"
    ) + 
    # Then the regression lines for each group
    geom_line(stat = "smooth", method = "lm", se = FALSE, alpha = 0.5) + 
    labs(title = "Individual trends vs overall trend") + 
    theme_minimal()

box::use(
    ggplot2[
        ggplot, aes, geom_histogram, geom_line, after_stat, 
        scale_fill_viridis_c, scale_color_viridis_c, 
        labs, theme_minimal, theme, margin, 
        element_text
    ]
)

ggplot(mtcars, aes(x = mpg)) +
    geom_histogram(
        aes(
            y = after_stat(density),  
            fill = after_stat(density > 0.05)  
        ),
        bins = 30,
        color = "white",
        linewidth = 0.2
    ) +
    geom_line(
        stat = "density", 
        mapping = aes(
            y = after_stat(density),  
            color = after_stat(density > 0.05)  
        ), 
        show.legend = FALSE
    ) + 
    scale_fill_viridis_c(option = "plasma") + 
    scale_color_viridis_c(option = "plasma") + 
    labs(
        title = "Histogram with after_stat()",
        subtitle = "Y-axis shows density, fill color shows count",
        x = "Miles per gallon",
        y = "Density",
        fill = "Count"
    ) +
    theme_minimal(base_size = 14) +
    theme(
        plot.title = element_text(face = "bold"),
        legend.position = "right",
        plot.margin = margin(15, 15, 15, 15)
    )

box::use(
    ggplot2[ggplot, aes, geom_point, labs, scale_color_identity],
    dplyr[if_else]
)

ggplot(mtcars) + 
    aes(x = wt, y = mpg, color = I(if_else(mpg > 20, "darkgreen", "darkred"))) + 
    geom_point(size = 4) +
    scale_color_identity(
        guide = "legend", 
        labels = c("Low MPG", "High MPG"),
        breaks = c("darkred", "darkgreen")
    ) +
    labs(color = "Efficiency")

box::use(
    ggplot2[ggplot, aes, geom_point, labs, theme_minimal]
)

ggplot(mtcars, aes(x = wt, y = mpg)) +
    geom_point(
        aes(
            size = cyl,
            shape = cut(
                hp, 
                breaks = c(0, 100, 200, 400), 
                labels = c("Low", "Medium", "High")
            )
        ), 
        alpha = 0.7
    ) +
    labs(size = "Horsepower", shape = "Cylinders") +
    theme_minimal()

8 My argument regarding ggplot2 implementation

I’ve argued with Python users who say “there’s {plotnine} in Python and you have all the features in ggplot2 and you can’t look back.” — it’s not even close. I maintain that it’s just a pale imitation: you can’t really fully replicate this package. It’ll take a lot to FULLY replicate it, and it’s TOO complicated for Python to replicate.

The thing is, R in general is functional; the arguments are lazily evaluated by default and similar to Lisp macros, where the AST can be transformed/modified during compile-time—you can pretty much manipulate and inspect expressions before they’re evaluated, giving you metaprogramming capabilities that are first-class citizens in R.

box::use(
    ggplot2[ggplot, aes, geom_point, geom_smooth]
)

ggplot(mtcars, aes(x = mpg, y = hp * 2, color = factor(cyl))) +
    geom_point() +
    geom_smooth(aes(y = hp * 2), method = "lm")

from plotnine import ggplot, aes, geom_point, geom_smooth

(ggplot(mtcars, aes(x = 'mpg', y = 'hp * 2', color = 'factor(cyl)')) +
    geom_point() +
    geom_smooth(aes(y = 'hp * 2'), method = 'lm')) 

9 Resources

• ggplot2 Official Documentation
• ggplot2 Aesthetic specification

1 Why “careful”?

As you read in the title, why “careful”? Using tidyverse, or any frameworks for data manipulation out there, is fine. Most introductory tidyverse tutorials teach you how to write code with it. Very few teach you when the nice-looking code becomes dangerous or unnecessarily painful six months later.

This short guide focuses on the second part — the things that experienced people wish they had known earlier.

Before we start, here are the packages I’ll be using in this post:

1. Packages that are tidyverse. This includes the following specifics:

   • dplyr
   • tidyr
   • readr
   • lubridate
   • purrr

2. Packages that are not tidyverse. This includes the following specifics:

   • box
   • numberize
   • broom
   • {e1071}

And they will be used much later on.

2 Tidyverse is not just unquoted column names

You thought the reason why tidyverse framework differ from other frameworks, e.g. Pandas, Polars, or data.table, comes from accepting “columns” argument not being quoted? I mean, yes, and data.table does this as well, although the syntax and APIs are not completely the same. That’s not what I am trying to convey here.

# This fails R CMD check in a package
mtcars |> dplyr::mutate(new = mpg / wt) |> head(5)
#>>                    mpg cyl disp  hp drat    wt  qsec vs am gear carb      new
#>> Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4 8.015267
#>> Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4 7.304348
#>> Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1 9.827586
#>> Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1 6.656299
#>> Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2 5.436047

# `R CMD CHECK` friendly
mtcars |> dplyr::mutate(new = .data$mpg / .data$wt) |> head(5)
#>>                    mpg cyl disp  hp drat    wt  qsec vs am gear carb      new
#>> Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4 8.015267
#>> Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4 7.304348
#>> Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1 9.827586
#>> Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1 6.656299
#>> Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2 5.436047

Why did I tell you this? Let’s say you’re planning to use this package as one of your package dependencies. Unfortunately, R CMD CHECK will fail and will complain if you are using bare column names, so instead use .data$colname.

Update: Thanks to u/Sufficient_Meet6836 from this thread for pointing it out, and Hadley Wickham respectfully recommends .data[[colname]], rather than .data$colname.

Here’s why:

1. .data$colname will throws an error once used in a closure:

   new_col = function(data, var1, var2) {
       data |> 
           dplyr::mutate(new = .data$var1 / .data$var2) |> 
           head(5)
   }
   
   mtcars |> new_col(mpg, wt)

2. .data[[colname]] with ensym() is less painful and accepts both bare columns and stringed columns:

   new_col = function(data, var1, var2) {
       box::use(rlang[ensym, as_string], dplyr[mutate])
       var1 = as_string(ensym(var1))
       var2 = as_string(ensym(var2))
   
       data |> 
           mutate(new = .data[[var1]] / .data[[var2]]) |> 
           head(5)
   }
   
   mtcars |> new_col(mpg, wt)
   mtcars |> new_col('mpg', 'wt')

Another method is the use of { name } when used in a different context. The most optimal application is when you’re writing functions that takes column names as arguments.

my_summarise1 = function(data, col) {
    data |> 
        dplyr::summarise(mean_col := mean({{ col }}, na.rm = TRUE))
}

iris |> my_summarise1(Sepal.Length)

  mean_col
1 5.843333

Here’s more complex as it uses !! (pronounced as “bang-bang”) operator:

my_summarise2 = function(data, col) {
    box::use(
        dplyr[summarise],
        glue[glue],
        rlang[ensym, as_string]
    )
    
    capt_col = ensym(col)
    name_col = glue("mean_{as_string(capt_col)}")
    
    data |> 
        summarise(!!name_col := mean({{ col }}, na.rm = TRUE))
}

iris |> my_summarise2(Sepal.Length)

  mean_Sepal.Length
1          5.843333

Click this link for more details.

3 Don’t just straight data imports

Do not just import data (not limited to CSVs), most especially when the data wasn’t collected in one pattern. For example, a CSV data that has “date” column(s), where it doesn’t follow the right format, e.g. %Y-%m-%d (translated as “2005-01-29” for example). This is probably the #1 source of silent bugs in production data pipelines, especially if overlooked.

But first, let me tell you something:

box::use(readr[read_csv, cols, col_character])

# Dangerous (very common in real life)
read_csv("data.csv")

# Much safer minimum
read_csv(
    "data.csv",
    col_types = cols(
        .default = col_character(), # This force everything as text first 
    )
)

sample_csv = "
date, readings, x, y, z\n
\"Mar.23,2005\",0.02,3.3,1.1,5.7\n
\"Feb.26,2005\",0.15,4.5,5.0,1.9\n
\"Apr.5,2005\",0.2,7.2,2.8,5.2\n
\"May.12,2005\", 0.5,4.9,1.3,6.8
"

readr::read_csv(I(sample_csv))

# A tibble: 4 × 5
  date        readings     x     y     z
  <chr>          <dbl> <dbl> <dbl> <dbl>
1 Mar.23,2005     0.02   3.3   1.1   5.7
2 Feb.26,2005     0.15   4.5   5     1.9
3 Apr.5,2005      0.2    7.2   2.8   5.2
4 May.12,2005     0.5    4.9   1.3   6.8

box::use(
    readr[read_csv, cols, col_date]
)

read_csv(
    I(sample_csv),
    col_types = cols(
        date = col_date("%b.%d,%Y")
    )
)

# A tibble: 4 × 5
  date       readings     x     y     z
  <date>        <dbl> <dbl> <dbl> <dbl>
1 2005-03-23     0.02   3.3   1.1   5.7
2 2005-02-26     0.15   4.5   5     1.9
3 2005-04-05     0.2    7.2   2.8   5.2
4 2005-05-12     0.5    4.9   1.3   6.8

box::use(
    readr[read_csv, cols, col_character],
    dplyr[mutate], 
    lubridate[parse_date_time, as_date]
)

sample_csv = "
date, readings, x, y, z\n
\"Mar.23,2005\",0.02,3.3,1.1,5.7\n
\"Feb 26, 2005\",0.15,4.5,5.0,1.9\n
\"04-5-2005\",0.2,7.2,2.8,5.2\n
\"May12,05\", 0.5,4.9,1.3,6.8\n
\"Aug.17,2005\", 0.17,9.5,0.8,4.2
"

read_csv(
    I(sample_csv),
    col_types = cols(
        date = col_character()
    )
) |> 
    mutate(
        date = 
            parse_date_time(
                date,
                orders = c(
                    "b.d,Y",
                    "b d, Y",
                    "m-d-Y",
                    "bd,y"
                )
            ) |> 
            as_date()
    )

# A tibble: 5 × 5
  date       readings     x     y     z
  <date>        <dbl> <dbl> <dbl> <dbl>
1 2005-03-23     0.02   3.3   1.1   5.7
2 2005-02-26     0.15   4.5   5     1.9
3 2005-04-05     0.2    7.2   2.8   5.2
4 2005-05-12     0.5    4.9   1.3   6.8
5 2005-08-17     0.17   9.5   0.8   4.2

4 Use tidyverse aggressively for serious data cleaning

Now you know how easy it get when importing the data with messed up pattern structure. Use tidyverse for data cleaning for f sake, trust me. Seriously, if you can do data cleaning from other tools, e.g. Excel or Python-Pandas (I know some companies you are working with will choose them), tidyverse makes things much easier, conventional, readable, and maintainable (arguable). Some practitioners actually use tidyverse for modeling and resort to base R loops or hacky solutions for cleaning. Don’t be that person.

Let us demonstrate with the Messy HR Data dataset from this repository.

hr_data = readr::read_csv(
    "https://raw.githubusercontent.com/eyowhite/Messy-dataset/main/messy_HR_data.csv"
)

If you glimpse this data:

hr_data |> dplyr::glimpse()

Rows: 1,000
Columns: 10
$ Name                <chr> "grace", "david", "hannah", "eve", "grace", "jack"…
$ Age                 <chr> "25", "nan", "35", "nan", "nan", "nan", "nan", "40…
$ Salary              <chr> "50000", "65000", "SIXTY THOUSAND", "50000", "NAN"…
$ Gender              <chr> "Male", "Female", "Female", "Female", "Female", "O…
$ Department          <chr> "HR", "Finance", "Sales", "IT", "Finance", "Market…
$ Position            <chr> "Manager", "Director", "Director", "Manager", "Man…
$ `Joining Date`      <chr> "April 5, 2018", "2020/02/20", "01/15/2020", "Apri…
$ `Performance Score` <chr> "D", "F", "C", "A", "F", "F", "B", "C", "C", "A", …
$ Email               <chr> "email@example.com", "user@domain.com", "email@exa…
$ `Phone Number`      <chr> "nan", "123-456-7890", "098-765-4321", NA, "098-76…

You see how bad it is, even if you said it’s not “dirty enough” or similar. The reason why tidyverse makes things “conventional” because in R, missing values has its own unique representation value, denoted by NA, not through null, NULL, or NaN / nan.

The data cleaning in R with tidyverse is much easier than you thought

box::use(
    dplyr[mutate, across, everything, if_else, case_when],
    stringr[detect = str_detect, capitalize = str_to_sentence],
    lubridate[parse_date_time, as_date]
)

hr_data |> 
    mutate(
        across(
            everything(), 
            \(col) if_else(detect(col, "^(?i)nan$"), NA, col)
        ), 
        `Joining Date` = parse_date_time(
            `Joining Date`, c(
                "%B %d, %Y",
                "%Y/%m/%d", 
                "%m-%d-%Y",
                "%Y.%m.%d"
            )
        ) |> as_date(), 
        Salary = numberize::numberize(Salary),
        Name = capitalize(Name),
        Age = numberize::numberize(Age)
    )

# A tibble: 1,000 × 10
   Name      Age Salary Gender Department Position  `Joining Date`
   <chr>   <dbl>  <dbl> <chr>  <chr>      <chr>     <date>        
 1 Grace      25  50000 Male   HR         Manager   2018-04-05    
 2 David      NA  65000 Female Finance    Director  2020-02-20    
 3 Hannah     35  60000 Female Sales      Director  2020-01-15    
 4 Eve        NA  50000 Female IT         Manager   2018-04-05    
 5 Grace      NA     NA Female Finance    Manager   2020-01-15    
 6 Jack       NA  65000 Other  Marketing  Director  2019-03-25    
 7 Charlie    NA  50000 Male   Marketing  Clerk     2019-12-01    
 8 Grace      40  50000 Other  HR         Director  2019-03-25    
 9 Hannah     40  60000 Female Marketing  Manager   2020-01-15    
10 Eve        30     NA Other  Finance    Assistant 2020-02-20    
# ℹ 990 more rows
# ℹ 3 more variables: `Performance Score` <chr>, Email <chr>,
#   `Phone Number` <chr>

You see how easy is this? The fact that dplyr::mutate() accepts bare expressions together with its bare column, you can easily use other functions, like numberize::numberize() to translate bare word value, i.e. "thirty" in Age column and "SIXTY THOUSAND" in Salary column.

5 Use across() + for safe type / role-based transformations

You see how I clean the data above, right? I use one of tidyverse’s most underrated features: across() combined with <tidy-select> helpers, commonly used in “data masking” functions, where the operation is applied across multiple columns. This lets you apply transformations to multiple columns based on their type or name pattern, making your code both DRY and maintainable.

box::use(
    dplyr[mutate, across, where],
    stringr[str_trim, str_to_upper]
)

survey = tibble::tibble(
    resp_id = c(1, 2, 3),
    name = c("  alice  ", "BOB", "  Charlie  "),
    city = c("New York  ", "  Boston", "Chicago  "),
    rating = c(4.2, 3.8, 4.5)
)

survey |>
    mutate(
        across(where(is.character), str_trim),
        across(where(is.character) & !resp_id, str_to_upper)
    )

# A tibble: 3 × 4
  resp_id name    city     rating
    <dbl> <chr>   <chr>     <dbl>
1       1 ALICE   NEW YORK    4.2
2       2 BOB     BOSTON      3.8
3       3 CHARLIE CHICAGO     4.5

box::use(dplyr[mutate, across, starts_with])

financials = tibble::tibble(
    company = c("TechCo", "RetailCorp"),
    revenue_q1 = c(1200000, 850000),
    revenue_q2 = c(1350000, 920000),
    costs_q1 = c(800000, 600000),
    costs_q2 = c(850000, 640000)
)

financials |>
    mutate(
        across(
            starts_with("revenue"),
            \(x) x / 1e6,
            .names = "{.col}_M"
        )
    )

# A tibble: 2 × 7
  company    revenue_q1 revenue_q2 costs_q1 costs_q2 revenue_q1_M revenue_q2_M
  <chr>           <dbl>      <dbl>    <dbl>    <dbl>        <dbl>        <dbl>
1 TechCo        1200000    1350000   800000   850000         1.2          1.35
2 RetailCorp     850000     920000   600000   640000         0.85         0.92

box::use(dplyr[mutate, across, where, any_of])

measurements = tibble::tibble(
    sample_id = 1:4,
    temperature = c(98.6234, 99.1456, 97.8921, 98.4567),
    pressure = c(120.456, 118.234, 122.789, 119.123),
    ph_level = c(7.4123, 7.3987, 7.4256, 7.4089)
)

measurements |>
    mutate(
        across(
            where(is.numeric) & !any_of("sample_id"),
            round,
            digits = 2
        )
    )

# A tibble: 4 × 4
  sample_id temperature pressure ph_level
      <int>       <dbl>    <dbl>    <dbl>
1         1        98.6     120.     7.41
2         2        99.2     118.     7.4 
3         3        97.9     123.     7.43
4         4        98.5     119.     7.41

box::use(
    tidyselect[where, matches], 
    dplyr[summarise, across],
    tidyr[long = pivot_longer, wide = pivot_wider],
    purrr[compose, partial], 
    stats[sd, quantile],
    e1071[skewness, kurtosis]
)

iris |> 
    summarise(
        across(
            where(is.numeric), 
            list(
                n_missing = compose(sum, is.na), # Since compose() returns a function
                mean = ~ mean(., na.rm = TRUE), 
                sd = \(col) sd(col, na.rm = TRUE), 
                min = min, 
                q1 = ~ quantile(., prob = 0.25, na.rm = TRUE), 
                md = \(col) median(col, na.rm = TRUE), 
                q3 = ~ quantile(., prob = 0.75, na.rm = TRUE), 
                max = \(col) max(col, na.rm = TRUE),
                skew = skewness,
                kurt = partial(kurtosis, na.rm = TRUE) # Since partial() returns a function
            ), 
            
            .names = "{.col}..{.fn}"
        )
    ) |> 
    long(
        cols = matches("\\.\\."), 
        names_pattern = "(.+)\\.\\.(.+)",  
        names_to = c("variable", "statistic"),
        values_to = "est"
    ) |> 
    wide(
        names_from = statistic,
        values_from = est
    ) 

# A tibble: 4 × 11
  variable     n_missing  mean    sd   min    q1    md    q3   max   skew   kurt
  <chr>            <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>  <dbl>  <dbl>
1 Sepal.Length         0  5.84 0.828   4.3   5.1  5.8    6.4   7.9  0.309 -0.606
2 Sepal.Width          0  3.06 0.436   2     2.8  3      3.3   4.4  0.313  0.139
3 Petal.Length         0  3.76 1.77    1     1.6  4.35   5.1   6.9 -0.269 -1.42 
4 Petal.Width          0  1.20 0.762   0.1   0.3  1.3    1.8   2.5 -0.101 -1.36 

6 Joins flavor in dplyr

Since the stable version (1.1.x) release of dplyr, specification of joins is now more flexible, more explicit, and more “controlled” using dplyr::join_by(). While the new API provides clearer complex “join” tasks, this also makes the operation less error-prone.

box::use(dplyr[left_join, join_by, group_by, slice_max, ungroup])

purchases = tibble::tibble(
    customer_id = 1:8,
    total_spent = c(45, 280, 520, 95, 1200, 180, 650, 35)
)

segments = tibble::tibble(
    segment = c("Bronze", "Silver", "Gold", "Platinum"),
    min_spend = c(0, 100, 500, 1000)
)

purchases |>
    left_join(segments, by = join_by(total_spent >= min_spend)) |>
    group_by(customer_id) |>
    slice_max(min_spend, n = 1, with_ties = FALSE) |>
    ungroup()

# A tibble: 8 × 4
  customer_id total_spent segment  min_spend
        <int>       <dbl> <chr>        <dbl>
1           1          45 Bronze           0
2           2         280 Silver         100
3           3         520 Gold           500
4           4          95 Bronze           0
5           5        1200 Platinum      1000
6           6         180 Silver         100
7           7         650 Gold           500
8           8          35 Bronze           0

box::use(
    dplyr[tbl = tibble, inner_join, join_by, between, transmute],
    lubridate[as_date]
)

transactions = tbl(
    tx_id = 1:6,
    tx_date = as_date(c(
        "2024-01-20", "2024-02-15", "2024-03-25",
        "2024-04-10", "2024-05-15", "2024-06-05"
    )),
    amount = c(150, 200, 180, 220, 175, 190)
)

promos = tbl(
    promo = c("New Year", "Spring Sale", "Summer Blast"),
    start = as_date(c("2024-01-01", "2024-03-01", "2024-05-01")),
    end = as_date(c("2024-01-31", "2024-04-30", "2024-06-30")),
    discount = c(0.15, 0.10, 0.20)
)

transactions |>
    inner_join(
        promos,
        by = join_by(between(tx_date, start, end))
    ) |>
    transmute(
        tx_id,
        tx_date,
        promo, 
        amount,
        discount, 
        savings = amount * discount
    )

# A tibble: 5 × 6
  tx_id tx_date    promo        amount discount savings
  <int> <date>     <chr>         <dbl>    <dbl>   <dbl>
1     1 2024-01-20 New Year        150     0.15    22.5
2     3 2024-03-25 Spring Sale     180     0.1     18  
3     4 2024-04-10 Spring Sale     220     0.1     22  
4     5 2024-05-15 Summer Blast    175     0.2     35  
5     6 2024-06-05 Summer Blast    190     0.2     38  

SELECT
    tx_id,
    tx_date,
    promo,
    amount,
    discount,
    amount * discount AS savings
FROM (
    SELECT t.*, p.*
    FROM transactions AS t
    INNER JOIN promos AS p
        ON (
            t.tx_date >= p.start AND
            t.tx_date <= p.end
        )
) AS new_table

7 Advanced tidyverse Tricks for Grouped Modeling

What’s so great about tidyverse is while easy to use, there are a lot of things you can take advantage of, such as the code below — it’s far more flexible than you thought. In the example below, I take advantage of dplyr::reframe() to generate a new data frame with a variable number of rows directly from grouped operations on the original data.

In this case, we’re exploring the linear relationship between vehicle weight (wt) and fuel efficiency (mpg) in the classic {mtcars} dataset, stratified by the number of cylinders (cyl). The code below computes key summary statistics — including regression coefficients, Pearson correlation, R-squared, and adjusted R-squared—for each cylinder group.

box::use(dplyr[group_by, reframe])

mtcars |> 
    group_by(cyl) |> 
    reframe(
        {         
            box::use(
                stats[linear_reg = lm, coef, pearson_r = cor], 
                purrr[imap_dfc, set_names], 
                tibble[tbl = tibble]
            )
            
            model = linear_reg(mpg ~ wt)
            coefs = coef(model)
            coef_table = imap_dfc(coefs, \(bi, nm) {
                result = tbl(bi)
                set_names(result, nm)
            })
            
            corr = pearson_r(wt, mpg)
            
            test = summary(model)
            tbl(
                coef_table, 
                corr = corr, 
                rsq = test$r.squared,
                adj_rsq = test$adj.r.squared
            )
        }
        
        # , .by = cyl
    )

# A tibble: 3 × 6
    cyl `(Intercept)`    wt   corr   rsq adj_rsq
  <dbl>         <dbl> <dbl>  <dbl> <dbl>   <dbl>
1     4          39.6 -5.65 -0.713 0.509   0.454
2     6          28.4 -2.78 -0.682 0.465   0.357
3     8          23.9 -2.19 -0.650 0.423   0.375

The code cleverly uses dplyr::reframe() with an anonymous block {}to run a complete mini-analysis per cylinder group in mtcars dataset. Inside the block, it loads just the needed functions via box::use(), fits a linear model of mpg ~ wt on the current group, extracts coefficients into a tidy one-row table, computes Pearson correlation, and grabs and adjusted from the model summary. Everything is then combined into a single row per group. This compact, namespace-safe pattern lets you perform fairly complex grouped modeling without cluttering the environment with temporary objects.

8 Closing Remarks and Resources

Those are all my “simple annotation” on your fussy data work, potentially (hoping) will help in your future jobs.

• R for Data Science
• Box: Placing module system into R
• box: Write Reusable, Composable and Modular R Code
• Statistical Rethinking
• dplyr 1.1.0 blog post - Deep dive on join_by()
• Programming with dplyr - When to use .data$ and { }

1 The Problem with Throwaway Code

Why bother yourself writing reusable codes? R is so behind in terms of reusability and maintainability of the codes written, and I don’t like the fact that there are piles of garbage codes in the wild — although other languages, even Python, is guilty at this one.

If you’ve been working with R for any length of time, you’ve probably encountered (or written) code that looks something like this:

analysis_final_v3_ACTUAL.R

library(tidyverse)

records <- read_csv("records.csv") # Directly import CSV files into R

# Modify the records a bit
records$date <- as.Date(records$date) 
records <- records %>% filter(!is.na(value))

# Calculate some statistics
mean_value <- mean(records$value)
sd_value <- sd(records$value)

# Then the usual records viz
ggplot(records, aes(x = date, y = value)) +
    geom_line() +
    theme_minimal()

This script works, yes. It does what you need it to do right now, yes. But what happens when you need to:

• Run the same analysis on a different dataset?
• Share this code with a colleague?
• Come back to this code in six months?
• Use these calculations in another project?

You end up copying and pasting, making small modifications, and before you know it, you have analysis_v1.R, analysis_v2.R, analysis_final.R, analysis_final_ACTUAL.R, and analysis_final_ACTUAL_USE_THIS_ONE.R scattered across your projects.

This is throwaway code. It solves an immediate problem but creates long-term technical debt.

2 What Makes Code Reusable?

Reusability of the code is about writing code with intention, structure, and foresight. It shouldn’t be limited about writing functions (though that helps).

Here are the key characteristics:

1. Clear separation of concerns - Each piece of code should do one thing well. Data loading, cleaning, analysis, and visualization should be separate operations that can be mixed and matched.
2. Minimal dependencies - Your code should depend on what it actually needs, not load 20 packages “just in case”. This helps for better long-term maintainability, and easier to understand.
3. Explicitness - Functions and codes in general should have clear inputs and outputs. No hidden dependencies on global variables or less mysterious side effects.
4. Documentation - Do not just indicate the code with comments, I recommend writing an actual documentation that explains what the code does, what it expects, and what it returns.

3 The Cost of Throwaway Code

Let me be blunt: throwaway code is expensive. Not in terms of money (though that too), but in terms of time, mental energy, and opportunity cost.

4 Reusability in native R

So how do we write reusable code in R? R offers few functionalities, but they are too fragile and suffered with numbers of limitations. I can’t recommend them enough, even for new R users.

library(dplyr)

data = data %>% 
    filter(!is.na(value), value > 0)

filter.R

library(dplyr)

retain_positive_value = function(data, var) {
    data %>%
        filter(!is.na({{ var }}), {{ var }} > 0)
}

retain_positive_value(data, value)

source("./filter.R")

5 Enter {box} package

I already have a book dedicated to code reusability and module systems using {box} package, with discussions about it. In fact, in every blog post I write, I always use box (sometimes just ::) to qualify the imports, rather than loading an entire package and attaching all exports of that particular package.

The box package provides a “lightweight”, modern module system for R that gives you the benefits of packages without the overhead.

Example usage:

install.packages("box") 
install.packages('box', repos = 'https://klmr.r-universe.dev')

box::use(
    dplyr,                                          # Loading the package without attaching the names
    ./R/data_cleaning,                              # Loading an entire particular script for data cleaning from the root path
    etl = ./R/data_cleaning,                        # Same as above but the alias was provided
    ./R/data_cleaning[clean_data, validate_data]    # Loading some names from a particular script for data cleaning from the root path
)

With box, you get:

• Explicit imports: Only load what you need
• Namespace isolation: No pollution of global environment
• Module encapsulation: Clear boundaries between code
• Simple syntax: Easy to learn and use
• Hierarchical structure: Organize modules in nested directories

project/
├── R/
│   ├── __init__.R       # <------ This will mark `{./R}` folder as a module
│   ├── data_loading.R
│   ├── data_cleaning.R
│   ├── analysis.R
│   └── visualization.R
├── data/
├── output/
└── main.R

./R/analysis.R

box::use(
    dplyr[
        summarise, across, n, relocate, pick,
        cur_group_id, matches
    ],
    tidyr[pivot_longer, pivot_wider]
)

#' @export
summary_data = function(data, vars, .by = NULL) {
    mtcars |>
        summarise(
            grp_id = cur_group_id(), 
            n = n(),
            across(
                {{ vars }},
                list(
                    mean = \(x) mean(x, na.rm = TRUE),
                    median = \(x) median(x, na.rm = TRUE), 
                    q25 = \(x) quantile(x, 0.25, na.rm = TRUE),
                    q75 = \(x) quantile(x, 0.75, na.rm = TRUE), 
                    sd = \(x) sd(x, na.rm = TRUE),
                    cv = \(x) sd(x, na.rm = TRUE) / mean(x, na.rm = TRUE),
                    iqr = \(x) IQR(x, na.rm = TRUE),
                    mad = \(x) mad(x, na.rm = TRUE)  
                ), 
                .names = "{.col}..{.fn}"
            ),
            
            .by = {{ .by }}
        ) |> 
        pivot_longer(
            cols = matches("\\.\\."), 
            names_pattern = "(.+)\\.\\.(.+)",  
            names_to = c("variable", "statistic"),
            values_to = "est"
        ) |>
        pivot_wider(
            names_from = statistic,
            values_from = est
        ) |>
        relocate(n, .after = variable)
}

box::use(pander[ptb = pandoc.table])

mtcars |> 
    summary_data(vars = c(mpg, hp, wt), .by = cyl) |> 
    ptb()

------------------------------------------------------------------------
 cyl   grp_id   variable   n    mean    median    q25     q75      sd   
----- -------- ---------- ---- ------- -------- ------- ------- --------
  6      1        mpg      7    19.74    19.7    18.65    21     1.454  

  6      1         hp      7    122.3    110      110     123    24.26  

  6      1         wt      7    3.117   3.215    2.822   3.44    0.3563 

  4      2        mpg      11   26.66     26     22.8    30.4     4.51  

  4      2         hp      11   82.64     91     65.5     96     20.93  

  4      2         wt      11   2.286    2.2     1.885   2.622   0.5696 

  8      3        mpg      14   15.1     15.2    14.4    16.25    2.56  

  8      3         hp      14   209.2   192.5    176.2   241.2   50.98  

  8      3         wt      14   3.999   3.755    3.533   4.014   0.7594 
------------------------------------------------------------------------

Table: Table continues below

 
---------------------------
   cv       iqr      mad   
--------- -------- --------
 0.07362    2.35    1.927  

 0.1984      13     7.413  

 0.1143    0.6175   0.3632 

 0.1691     7.6     6.523  

 0.2533     30.5    32.62  

 0.2492    0.7375   0.5411 

 0.1695     1.85    1.557  

 0.2437      65     44.48  

 0.1899    0.4812   0.4077 
---------------------------

box::use(./R/analysis[summary_data])

mtcars |> 
    summary_data(vars = c(mpg, hp, wt))

box::use(
    # Import the module itself without attaching the names (access functions with summary$function_name)
    ./analysis, 
    # Import specific names
    ./analysis[summary_data, another_function], 
    # Attach all exported functions
    ./analysis[...]
)

./R/analysis.R

box::use(
    dplyr[
        summarise, across, n, relocate, pick,
        cur_group_id, matches
    ],
    tidyr[pivot_longer, pivot_wider]
)

#' Get summary data from numeric column
#' 
#' Calculate comprehensive summary statistics for numeric variables,
#' including measures of central tendency, dispersion, and spread.
#' 
#' @param data A data frame
#' @param vars Vector of columns
#' @param .by Optional grouping variable(s)
#' 
#' @return A data frame with summary statistics in long format
#' 
#' @examples 
#' mtcars |> summary_data(vars = c(mpg, hp, wt))
#' mtcars |> summary_data(vars = c(mpg, hp), .by = cyl)
#' 
#' @export
summary_data = function(data, vars, .by = NULL) {
    mtcars |>
        summarise(
            grp_id = cur_group_id(), 
            n = n(),
            across(
                {{ vars }},
                list(
                    mean = \(x) mean(x, na.rm = TRUE),
                    median = \(x) median(x, na.rm = TRUE), 
                    q25 = \(x) quantile(x, 0.25, na.rm = TRUE),
                    q75 = \(x) quantile(x, 0.75, na.rm = TRUE), 
                    sd = \(x) sd(x, na.rm = TRUE),
                    cv = \(x) sd(x, na.rm = TRUE) / mean(x, na.rm = TRUE),
                    iqr = \(x) IQR(x, na.rm = TRUE),
                    mad = \(x) mad(x, na.rm = TRUE)  
                ), 
                .names = "{.col}..{.fn}"
            ),
            
            .by = {{ .by }}
        ) |> 
        pivot_longer(
            cols = matches("\\.\\."), 
            names_pattern = "(.+)\\.\\.(.+)",  
            names_to = c("variable", "statistic"),
            values_to = "est"
        ) |>
        pivot_wider(
            names_from = statistic,
            values_from = est
        ) |>
        relocate(n, .after = variable)
}

box::use(./R/analysis)

box::help(analysis$summary_data)

6 Conclusion

I don’t know about you (who read this blog post), but writing reusable code isn’t about being pedantic or following rules for the sake of rules. It’s about respecting your future self, your colleagues, and the craft of programming.

R suffers a lot of limitations in terms of reusability and maintainability. Many other programming languages’ users remarks R as being an odd one, and goes to say “it suffers for large projects”, and simply because R doesn’t have a right tool. Fortunately, I can’t thank box package enough by giving R users a modern, lightweight way to organize code into reusable modules without the overhead of creating full packages, similar to Python module system. It’s a middle ground that’s been missing from the R ecosystem.

Start small. Pick one project and try organizing it with box. You’ll quickly see the benefits:

• Clearer code structure
• Easier maintenance
• Better collaboration
• Less time wasted on repetitive tasks

Your future self will thank you. And maybe, just maybe, we can reduce the amount of garbage code in the wild.

7 Resources

• {box} package documentation
• {box} GitHub repository
• Box: Placing module system into R
• R Packages book by Hadley Wickham - for when you’re ready to go beyond modules

1 Level 1: Manual Composition

At the most basic level, we compose functions by manually writing them out. This is the approach most R users start with, and it’s perfectly valid for many use cases.

normalize = function(x) {
    (x - mean(x)) / sd(x)
}

data = c(10, 20, 30, 40, 50)
normalize(data)

[1] -1.2649111 -0.6324555  0.0000000  0.6324555  1.2649111

sqrt2 = function(x) x^0.5

sapply(1:5, sqrt2)

[1] 1.000000 1.414214 1.732051 2.000000 2.236068

sapply(1:5, function(x) x^0.5)

[1] 1.000000 1.414214 1.732051 2.000000 2.236068

sapply(1:5, \(x) x^0.5)

[1] 1.000000 1.414214 1.732051 2.000000 2.236068

quote(\(x) x^2)

function(x) x^2

2 Level 2: Using Higher-order Functions

How about we comprises the existing functions and give birth to another function? We can easily make that with purrr::compose(). How about we create a function out of the function? That’s function operators in action.

Higher-order functions can return new functions by combining or modifying existing ones, reducing manual code writing.

transform = function(x) {
    sqrt(mean(log(x)))
}

transform(1:5)

[1] 0.9785184

purrr::compose(sqrt, mean, log)(1:5)

[1] 0.9785184

1:5 |> log() |> mean() |> sqrt()

[1] 0.9785184

purrr::compose(sqrt, mean, log, .dir = "forward")(1:5)

[1] 0.5166883

1:5 |> sqrt() |> mean() |> log()

[1] 0.5166883

transform_forward = purrr::compose(sqrt, mean, log, .dir = "forward")
transform_forward(1:5)

[1] 0.5166883

1:5 |> sqrt() |> mean() |> log()

[1] 0.5166883

divide_by_100 = function(x) {
    x / 100
}

divide_by_100(500)

[1] 5

divide_by_100 = purrr::partial(`/`, e2 = 100)
divide_by_100(500)

[1] 5

with_logging = function(f) {
    function(...) {
        cat("Calling function with args:", paste(list(...), collapse = ", "), "\n")
        res = f(...)
        cat("Result:", res, "\n")
        res
    }
}

logged_sqrt = with_logging(sqrt)
logged_sqrt(16)

Calling function with args: 16 
Result: 4 

[1] 4

def with_logging(f):
    def call(*args):
        args_str = ", ".join(str(a) for a in args)
        print(f"Calling function with args: {args_str}")
        res = f(*args)
        print(f"Result: {res}")
        return res
    
    return call


@with_logging
def logged_sqrt(x):
    return x ** 0.5
    
print(logged_sqrt(16))

Calling function with args: 16
Result: 4.0
4.0

memoize = function(f) {
    cache = new.env(parent = emptyenv())
    
    function(...) {
        key = paste(list(...), collapse = "_")
        if (!exists(key, envir = cache)) {
            cache[[key]] = f(...)
        }
        cache[[key]]
    }
}

slow_function = function(x) {
    Sys.sleep(1) 
    x^2
}

fast_function = memoize(slow_function)
system.time(fast_function(5))

   user  system elapsed 
   0.00    0.00    1.01 

3 Level 3: Programmatic Approach

Swear, this is not an easy thing to do — you need at least better understanding on how to build / generate expressions in R, and this involves understanding metaprogramming in R — Yes, most of the part on manipulating ASTs.

The rlang::new_function() provides an API that programmatically constructs a function expression (yes, it does creates formals, body, and an environment — 3 main components of R functions). Keep in mind that the construction of the function expression with rlang::new_function() happens in runtime.

box::use(
    rlang[new_function, expr, caller_env, call2]
)

square = new_function(
    args = list(x = expr()), 
    body = call2("{", quote(x^2)), 
    env = caller_env()
)

square(5)

[1] 25

box::use(rlang[pairlist2])

square2 = new_function(
    args = pairlist2(x = ), 
    body = expr(x^2), 
    env = caller_env()
)

square2(5)