Student Assistant/HiWi LLM scaling laws (m/f/d)

 

About Max Planck Institute for Intelligent Systems 

The Max Planck Institute for Intelligent Systems (MPI-IS) conducts research to advance intelligent systems that perceive, learn, and interact. Through fundamental research across computational, physical, and social dimensions, the institute shapes artificial intelligence and robotics to advance knowledge and benefit society.

At Social Foundations of Computation, the team builds scientific foundations for machine learning and artificial intelligence in the social world. To chart and implement a society’s norms and expectations, they start from concepts and work their way towards applications. Challenging existing problem formulations when necessary, they think through how the use of machine learning distributes societal resources and opportunity. Computational tools to critically evaluate - and possibly contest - algorithmic systems and their impacts are a key component of the work. The ultimate goal is to promote a positive role of artificial intelligence in society.

 

About the project

The team is  building free-form question-answering datasets to study whether answer-matching and generative scoring yield smoother, more predictable scaling of LLM capabilities than multiple-choice metrics. The work spans data curation, validating correctness of benchmarks, and ambiguity audits—with lightweight tooling to run evaluations and visualize scaling behavior.

Relevant Papers:

• Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
• Answer Matching Outperforms Multiple Choice for Language Model Evaluation
• Pretraining Scaling Laws for Generative Evaluations of Language Models

 

Roles & Responsiblities

Your responsibilities may include:

• Build on top of existing codebase.
• Convert existing MCQ datasets (across diverse domains) to freeform format using language models.
• Run large-scale evaluations across different model sizes (pretrained checkpoints), analyse results, and chart scaling curves.
• Optional: Make a small web UI for browsing responses of models and verifying answer correctness manually if needed.
• Contribute to a project that is planned for conference submission.

 

Your Profile 

Requirements:

• Good coding skills in Python
• Experience with data processing (pandas, regex, jsonl), version control (Git), and LLM tooling (vLLM).
• Have solid experimental rigor: logging, ablations, seeds, and ensuring reproducibility
• Nice-to-have: Familiarity with pretraining, scaling laws, MCQ benchmarks, evaluation/metrics, annotation pipeline design.

Most importantly, MPI-IS is looking for people who can iterate fast and use AI-copilot coding tools like Cursor/Claude Code/etc. to significantly speed up implementation time/learn new concepts on the fly and get results more quickly.

 

The Offer

Students without a Bachelor’s degree will earn €13,80 per hour. Students holding a Bachelor’s degree will be paid €14,40 per hour. An initial contract will be offered for up to six months. Please do not hesitate to apply anytime, if you are interested in working with MPI-IS. Contracts are given per semester.

 

Application

To apply, please upload a motivational letter, a CV an references as one PDF file to the application portal. You are also welcome to send links to code or writing (GitHub, papers, blog).

Any questions regarding the position should be forwarded to Nikil Chandak at nikil.chandak@tuebingen.mpg.de.

If you are excited about building the datasets that make LLM progress more realistically measurable and predictable, MPI-IS would love to hear from you!

The posting is open until filled.

 

The Max Planck Society is committed to increasing the number of individuals with disabilities in its workforce and therefore encourages applications from such qualified individuals. The Max Planck Society strives for gender equality and diversity. Furthermore, the Max Planck Society seeks to increase the number of women in its workforce in those areas where they are underrepresented and therefore explicitly encourages women to apply.