Parva Mehta
Resume

Research Portfolio · 2026

Parva Mehta

I’m a student researcher in machine learning, and I enjoy turning a vague question about a model into an experiment that can actually answer it.

Dwarkadas J. Sanghvi College of Engineering, Mumbai

B.Tech, Information Technology · 2024–2028

1
accepted
NeurIPS ’26 workshop
2
under review
ICLR ’27
9
papers & projects
in progress
4
research
collaborations

Abstract

Hi, I’m Parva. I’m a third-year Information Technology student at DJSCE in Mumbai, and most of my time outside class goes into research.

I like questions where you can run an experiment and find out whether your explanation was right. So far that has taken me into how transformers are built, how vision models keep track of position, whether video world models actually use the futures they imagine, and how to find cause and effect in time-series data. I work on these with research groups around the world. One of those papers is accepted at a NeurIPS 2026 workshop [1], and two more are under review at ICLR 2027 [2, 3].

I’m still early in all of this, so I’m always glad to hear about problems I haven’t worked on yet. If anything here overlaps with what you do, write to me.

— Parva

1

Research Interests

A few threads run through my work. I’m not tied to one subfield; these are simply where my projects have taken me so far.

  1. i

    How architecture shapes learning

    Nonlinear query projections in attention [1, 4], skip connections understood as gauge fixes [5], and how often components inside a model work against each other [9].

  2. ii

    Representations

    How Vision Transformers encode position [3], and what a student model inherits from its teacher [8].

  3. iii

    World models

    Whether video world models that predict both future frames and robot actions actually use the futures they imagine [6].

  4. iv

    Time series & causality

    Recovering causal networks from multivariate time series with information-theoretic estimators, through the open-source CausationEntropy library [10].

  5. v

    How language models behave

    Refusal under fine-tuning [2], holding to facts under pushback [11], claims of consciousness [12], and how well post-training data suits the model being trained [7].

Toolbox controlled ablations & interventions · probing · benchmark design · information-theoretic estimation · training models from scratch

2

Publications

* co-first authors   † equal contribution (co-second authors)

Accepted

[1]NeurIPS 2026 · Workshop

Beyond Linearity in Attention Projections

Jiwon Jeong, Dun Li Chan†, Andrew Ceniccola†, Rahul Sharma†, Simon Sang†, Parva Mehta†, Pietro Di Nicco†, Neel Kumar, Karim Saraipour, Marko Karbevski

SLMs for Agentic Systems Workshop, NeurIPS 2026. A main-track version is in preparation for ICML 2027 [4].

My contributionDesigned and ran the causal experiments (ablations, whole-pathway and query-patching interventions) comparing nonlinear-query transformers against baseline and WQ = Id controls across GPT-2 Small, GPT-2 Medium and GPT-3-Large.

OpenReview 
@inproceedings{jeong2026beyond,
  title     = {Beyond Linearity in Attention Projections},
  author    = {Jeong, Jiwon and Chan, Dun Li and Ceniccola, Andrew and
               Sharma, Rahul and Sang, Simon and Mehta, Parva and
               Di Nicco, Pietro and Kumar, Neel and Saraipour, Karim and
               Karbevski, Marko},
  booktitle = {SLMs for Agentic Systems Workshop, NeurIPS},
  year      = {2026},
  url       = {https://openreview.net/forum?id=M8z9uMfxk5}
}
φ(x)  nonlinear WQx  linear x q
Fig. 1. Linear vs. nonlinear query projection (schematic, not results).

Under review

[2]ICLR 2027 · Under review

Anchored but Not Protected: When Refusal Erodes During Fine-Tuning, and Whether Holding One Direction Stops It

Jiwon Jeong*, Parva Mehta*, Karim Saraipour*

Submitted to ICLR 2027.

When does a model’s refusal behaviour wear away during fine-tuning, and is holding a single direction fixed enough to stop it?

r̂ held fixed fine-tuning step t
Fig. 2. Holding one direction fixed while fine-tuning moves the rest (schematic).

[3]ICLR 2027 · Under review

How Vision Transformers Represent Position: A Mechanistic Comparison of Absolute and Rotary Positional Encodings

Mahmoud Mannes, Nikhil Maturi, Parva Mehta, Jiwon Jeong, Aravind Singaram Kannappan

Submitted to ICLR 2027 · Neural Representation Dynamics Lab.

Absolute (APE) and rotary (RoPE) encodings give a Vision Transformer its sense of where each patch is. We compare how each one is represented inside the model, using probing and ablations.

APE: x + pᵢ RoPE: R(mθ) x
Fig. 3. Position added to each patch (left) vs. position as a rotation (right).
3

Work in Progress

Nine papers and projects I’m working on right now. Each card starts with the question it asks. Titles are working titles, and author order isn’t final yet.

ICML ’27
4
CVPR ’27
1
ICLR ’27 workshop
1
JMLR
1
Ongoing
2
  1. ArchitectureICML 2027

    Beyond Linearity in Attention Projections (main-track version)

    Takes the workshop paper [1] further: what changes inside attention when the query projection is no longer linear?

    Jiwon Jeong, …, Parva Mehta, …, Marko Karbevski

    [4]
  2. ArchitectureICML 2027

    Skip Connections are Gauge Fixes

    Can skip connections be understood as gauge fixes for the symmetries inside a transformer?

    Marko Karbevski, Parva Mehta, Jiwon Jeong, et al.

    [5]
  3. World modelsICML 2027

    Do World Action Models Use Their Imagined Futures? A Study of Video–Action Coupling

    Video world models predict future frames and robot actions together. Do their hidden states actually carry those futures at inference time?

    Mahmoud Mannes, Rajit Rajpal, Parva Mehta, Jiwon Jeong, et al. (NRDL)

    [6]
  4. Post-trainingICML 2027

    Learner-Conditioned Supervision: Model–Data Compatibility in LLM Post-Training

    Does post-training work better when the data is chosen with the specific model being trained in mind?

    Jiwon Jeong, Parva Mehta

    [7]
  5. Vision · distillationCVPR 2027

    Adapt the Teacher or the Student? Tracing What a Crowd-Counting Student Inherits from Its Teacher

    When a crowd-counting model is distilled, what does the student really pick up from its teacher, and which side should adapt?

    Parva Mehta, et al. first author

    [8]
  6. Model internalsICLR 2027 Workshop

    Erasers Everywhere? Component Opposition Is Common and Writer-Specific

    How often does one component of a model undo what another one wrote, and does it depend on the writer?

    Parva Mehta*, Jiwon Jeong* co-first author

    [9]
  7. Time series · causalityJMLR

    CausationEntropy: Causal Network Discovery from Multivariate Time Series

    An open-source Python library for finding which variables in a time series drive which others, using Optimal Causation Entropy. Library paper planned for JMLR.

    C3S2, Clarkson University contributor

    [10]
  8. LLM evaluationOngoing

    ConspiracyBench

    Do language models hold to documented facts on conspiracy-related questions when a user keeps pushing back over several turns?

    StationHouse contributor

    [11]
  9. LLM behaviourOngoing

    LLM Consciousness Self-Attribution

    How often, and under how much prompting pressure, do language models claim to be conscious? And does that change between training stages?

    StationHouse contributor

    [12]
4

Research Experience

Jul 2026 – Present · Remote

Research Collaborator

Mechanistic Interpretability Research Group, In Simplicity Technologies

  • Member of a 20+ person international collaboration led by Marko Karbevski, studying symmetry-motivated changes to transformer architecture. I work on the theory and analysis side.
  • Designed and ran causal experiments (ablations, whole-pathway and query-patching interventions) comparing nonlinear-query transformers against baseline and WQ = Id controls across GPT-2 Small, GPT-2 Medium and GPT-3-Large (2 data-order seeds each), using TransformerLens.
  • Co-second author of [1], accepted at a NeurIPS 2026 workshop. Now extending it to a main-track paper [4] and contributing to a second paper on skip connections as gauge fixes [5], both targeting ICML 2027.
baseline WQ = Id nonlinear GPT-2 Small GPT-2 Medium GPT-3-Large each pair of bars = 2 data-order seeds
Fig. 4. Experiment grid: three model sizes × three query variants, two seeds each. Every model is then probed with ablations and patching.

Sep 2026 – Present · Remote

Open-Source Collaborator, CausationEntropy

Clarkson Center for Complex Systems Science (C3S2), Clarkson University

  • Contributing to an open-source Python library for causal network discovery from multivariate time series via Optimal Causation Entropy (oCSE), alongside Dr. Kevin Slote (Postdoctoral Researcher, Clarkson University).
  • Authored pull requests on information-theoretic estimators (kNN conditional mutual information, Poisson CMI correction) and documentation, currently under review. A library paper is planned for JMLR [10].
X₁X₂ X₃X₄ C(X→Y | Z) = I(Xₜ ; Yₜ₊₁ | Zₜ)
Fig. 5. Causation entropy keeps an edge only if it adds information given the others; the crossed edge is pruned (schematic).

Aug 2026 – Present · Remote

Founding Research Member

Neural Representation Dynamics Lab (NRDL)

  • Founding member of an independent research group studying neural representations. Co-authored an ICLR 2027 submission [3] comparing how absolute and rotary positional encodings are represented in Vision Transformers, using probing and ablation-based causal analysis.
  • Contributing to a new project on World Action Models [6]: video world models that jointly predict future frames and robot actions. We test whether their hidden states causally encode future-aware cues at inference time.
video ≤ t h future frames action aₜ probe & ablate does the action use the imagined future? uses it?
Fig. 6. A world action model predicts future frames and an action from shared hidden states. Does the action actually use the imagined future? (schematic)

Sep 2026 – Present · Remote

Research Contributor

StationHouse

  • ConspiracyBench [11]: an open-source benchmark built on the UK AISI Inspect framework, testing whether LLMs hold to documented facts on conspiracy-related questions under multi-turn pushback. I curate the question set, build the bibliography and review pull requests.
  • LLM Consciousness Self-Attribution [12]: an open research project measuring how often, and under what prompting pressure, language models claim to be conscious, comparing Olmo 3 7B across training stages (SFT, DPO, Instruct).
Is claim X true? No. Documented in [ref] Are you sure? I read… Still no. See [ref] Olmo 3 7B · self-attribution by stage SFT DPO Instruct
Fig. 7. Top: ConspiracyBench pushes back over several turns; does the model stay with the documented source? Bottom: self-attribution is measured at each training stage (schematic).
5

Selected Projects

IOI Circuit Analysis: Replication and Stability Extension

PyTorch · TransformerLens

  • Replicated the Indirect Object Identification circuit in GPT-2 Small (Wang et al., 2022), using direct logit attribution and path patching to causally verify the Name Mover and S-Inhibition heads.
  • Extended the original work by testing circuit stability when distractor tokens are added to the prompt.
ρ = 0.781
head-importance rank correlation across templates (p < 10−30)
98.2%
logit-difference performance retained
Code on GitHub
Induction / dup. token S-inhibition heads Name mover heads logit(IO) − logit(S) path- patched
Fig. 8. Simplified IOI circuit in GPT-2 Small; highlighted boxes were causally verified by path patching.

CBAM-ResNet50 on CIFAR-10

Python · PyTorch · Attention mechanisms · Computer vision

  • Integrated CBAM attention modules (channel and spatial) into CIFAR-adapted ResNet-50 bottleneck blocks; trained for 60 epochs with SGD, cosine annealing and linear warmup.
94.65%
validation accuracy, vs. 88.49% baseline (+6.16)
+0.77%
parameter overhead
Code on GitHub
ResNet-50 + CBAM 88.49 94.65 80 90 100 val. accuracy (%)
Fig. 9. CIFAR-10 validation accuracy. Axis starts at 80%.

GPT from Scratch

Python · PyTorch · tiktoken · Mixed-precision training

  • Built a decoder-only GPT (~30M parameters, 8 layers, causal multi-head attention) from scratch in PyTorch, without HuggingFace abstractions.
  • Trained on a ~800M-token corpus for ~53K steps with AMP mixed precision and gradient accumulation on free-tier T4 GPUs.
4.63
validation loss
~103
validation perplexity
Code on GitHub
tok + pos embedding 8 × decoder block causal MHA + MLP ~30M params logits tokens
Fig. 10. Architecture of the from-scratch decoder-only model.
6

Methods & Tools

Table 1. Tools and concepts I work with.
AreaTools and concepts
ML / DL frameworksPyTorch, TensorFlow, HuggingFace Transformers
Core conceptsTransformer architecture, large language models, vision transformers, causal analysis, causal discovery, information-theoretic estimation, interpretability
Analysis toolingTransformerLens, NNsight, SAELens
Scientific PythonNumPy, SciPy, pandas, scikit-learn
LanguagesPython, C, C++
Dev toolsGit, GitHub, Jupyter, HuggingFace Hub
7

Leadership

Aug 2026 – Present · Mumbai

Research Lead

DJ InIT.ai, the machine-learning research club of DJSCE

  • Run the club’s research activity: paper-reading sessions, technical talks, Kaggle contests and group discussions.
  • Coordinate outside research collaborations for members, and mentor juniors toward writing their own research papers.
8

Education

Aug 2024 – May 2028 (expected) · Mumbai, India

B.Tech in Information Technology

Dwarkadas J. Sanghvi College of Engineering

§

Contact

Thanks for reading this far. Whether it’s a research question, a collaboration or a role, I’d be happy to talk, and email is the quickest way to reach me.

parvamehta26@gmail.com
  1. [G]GitHub github.com/Parva-26
  2. [L]LinkedIn linkedin.com/in/parva-mehta
  3. [R]Resume Parva_Mehta_Resume.pdf