Measure whether an interpretability change improved real model behavior.
Mohamed A M Elansary, PhD — scientific multimodel evaluation, uncertainty quantification, and honest measurement for understanding and steering large models.
Evaluation under uncertainty
- Six-plus years of multimodel, multi-basin forecast experiments across hydroclimates on Linux/HPC.
- Compared statistical and physically based stacks, quantified uncertainty, and reported regime-dependent failure modes rather than a single flattering score.
- That is the measurement analogue of asking whether a new technique for understanding or steering a model changed real behavior.
Agent evaluation sets
- Production GPT, Claude, and Gemini agent workflows with retrieval, routing, tenant isolation, provenance, and regression evaluation sets at Vertexium, including a multi-tenant conversational receptionist.
- That maps to inspecting whether a change improved intended behavior. It is not Silico ownership or mechanistic-interpretability method authorship.
- A PhD or equivalent in a quantitative science is a posted requirement; this profile includes an Environmental Engineering PhD.
Proposed first contribution
For one interpretability or model-steering evaluation already in flight, define what understanding and steering mean as observable evidence versus a score or visualization that is easy to move. Write a small failure taxonomy: metric movement without a causal behavioral change; a visualization that looks informative but does not predict held-out failure; a tool that helps on a demo slice and fails under regime shift; uncertainty that collapses when observations are imperfect. Stand up a small evaluation set with provenance, compare simple baselines, attach uncertainty, and write a clear report before expanding the research prototype. This is a proposed measurement approach, not a claim of prior Goodfire-internal work, Silico ownership, invented metrics, or safety research.
Honest fit boundary
Mechanistic interpretability and the Goodfire product stack are a stretch. I have not trained sparse autoencoders, built interpretability agents, owned Silico, published mech-interp methods, or claimed Goodfire-internal work, and I do not invent metrics or safety research. The credible contribution is scientific multimodel evaluation, uncertainty quantification, production agent evaluation harnesses, scientific/HPC rigor, data pipelines, and clear technical writing.
Role and location
Research Scientist · San Francisco, CA · in person 5 days a week, with one company-wide remote week per month. Willing to relocate to San Francisco with a relocation package. Remote work is not asserted.
Posting compensation: “The expected salary range for this position is $200,000 - $400,000 USD”. · Official role posting