Senior Storage & Data Engineer
80%-100%, Lugano, fixed-term
The Swiss National Supercomputing Centre (CSCS) develops and operates a high-performance computing and data research infrastructure that supports world-class science in Switzerland. Its user laboratory is available to domestic and international researchers in academia, industry, and the business sector. The centre is operated by ETH Zurich and has offices at its data centre in Lugano and in Zurich.
For this position, the work location is either Lugano or Zurich. The contract is for two years.
Project Background
Storing petabytes is the easy part. The challenging aspect lies in everything between the moment data lands on disk and the moment a researcher—or a training job—can actually trust, find, and use it. Our parallel filesystems and object stores facilitate swift data movement, but they do not inform scientists where a dataset originated, which transformations produced it, or how to effectively process it without overloading the I/O subsystem. This gap—between raw bytes and usable, traceable, reproducible data—is where this role fits in.
You will work at both ends: the storage layer (throughput, integrity, tiering at multi-petabyte scale) and the data layer above it (lineage, provenance, discoverability, access patterns). If you've ever been frustrated by the idea that *"the data is on the cluster"* marks the end of the job rather than the beginning, this position may be for you.
Job Description
- Bridge ingestion and use. Design the pipelines and metadata that turn ingested data into something findable and consumable—catalogs, schemas, and access layers that reflect how training jobs and simulations actually read, not just where bytes reside.
- Make data traceable. Build lineage and provenance so any dataset, checkpoint, or result can be traced back to its inputs and transformations. Reproducibility is a first-class requirement.
- Tune for the workload. Optimize parallel filesystems (Lustre, GPFS) and object storage for the concurrency, small-file, and large-checkpoint patterns of distributed GPU training and HPC simulation.
- Operate at scale, safely. Design and run multi-petabyte storage with the integrity and availability that scientific work depends on—erasure coding, redundancy, hot-to-archival tiering.
- Automate everything. Deploy and scale storage and data services as code. Traditional infrastructure does not suffice at this scale.
- Make it observable. Instrument storage health, capacity trends, and pipeline performance so issues arise before impacting users.
- Translate. Convert real access patterns from domain scientists and ML engineers into technical requirements—advocating for data integrity whenever necessary.
An opening for two years is available for a project in the weather and climate domain, aimed at understanding and mitigating the impact of climate change. The initial contract could potentially be extended or even become permanent.
Profile
- A technical degree (CS, engineering) or equivalent experience that demonstrates a similar depth.
- Solid storage grounding: filesystems (block and object), performance tuning, and redundancy (RAID, erasure coding).
- Proficiency in Python and familiarity with automating infrastructure (Ansible, Terraform, or similar).
- A working understanding of how ML and scientific workloads consume data—billions of small files, large checkpoints, sharding—and the pitfalls of naive layouts.
- An informed perspective on data lineage, provenance, or reproducibility, ideally with tools you’ve used to enforce these principles.
What Helps You Stand Out
- Hands-on experience with parallel filesystems (Lustre, Spectrum Scale/GPFS) or distributed storage (Ceph, VAST).
- Familiarity with scientific data formats such as HDF5, Zarr, Parquet and their applicable use cases.
- Experience with object storage (S3) interfaced with ML frameworks (PyTorch, TensorFlow).
- Knowledge of orchestration tools (Kubernetes, Argo) and data movement tooling.
- Experience with data versioning/cataloguing (e.g., DVC, lakeFS, metadata catalog) and familiarity with FAIR data principles.
- CI/CD and provisioning experience with tools such as GitLab CI, HashiCorp Vault, or MAAS.
We understand that not every qualification may apply to you. What matters most is depth in either storage or data engineering, along with the curiosity to grow into the other area.
What You Get
- Access to hardware and scale you won't find in enterprise IT, coupled with unique challenges beyond standard vendor playbooks.
- Experience that directly facilitates published science and cutting-edge model training.
- Opportunity to shape how data is managed—not merely maintained—in an environment that prioritizes it.
Our Core Values
- Curiosity: You enjoy learning and understanding systems deeply.
- Openness: You collaborate effectively and value diverse perspectives.
- Courage: You tackle difficult or unfamiliar challenges head-on.
- Supportive: You are committed to helping colleagues and users succeed.
- Integrity: You act responsibly, reliably, and transparently.
Workplace
We are committed to building a diverse and inclusive engineering team and particularly encourage applications from groups underrepresented in tech. If you are technically proficient, curious, and eager to grow, we want to hear from you.
- Your job will have a significant impact: Become part of ETH Zurich, which not only supports your professional development but also actively contributes to positive change in society.
- Expect numerous benefits, including public transport season tickets, car sharing, access to a wide range of sports offered by ASVZ, childcare, and attractive pension benefits.
- Look forward to an engaging work environment filled with cultural diversity and various attractive offers.
- We value the diversity of our team and especially encourage women to apply to enhance our workforce.
We Value Diversity and Sustainability
ETH Zurich promotes an inclusive culture, ensuring equality of opportunity and valuing diversity. We nurture a cooperative environment that respects the rights and dignity of our staff and students. Sustainability is one of our core values, and we are continuously working towards a climate-neutral future.
Curious? So Are We.
Apply online using the form below. Please note that only applications matching the job profile will be considered.
For further information about CSCS, please visit our website. Questions regarding the position should be directed to Pim Witlox at witlox@cscs.ch (please, no applications).
About ETH Zürich
ETH Zurich is one of the world's leading universities specializing in science and technology. Renowned for our outstanding education, cutting-edge research, and direct transfer of new knowledge into society, our university attracts over 30,000 individuals from more than 120 countries who thrive in an environment that fosters independent thinking and inspires excellence. Located in the heart of Europe while forging connections worldwide, we collaborate to develop solutions for today's and tomorrow's global challenges.