DescriptionWe are seeking a skilled Data Scientist I to join our team in the SinAI Assurance Lab. The position will play a key role in Machine Learning Operations and be responsible for validation, monitoring, and governance of AI models across clinical workflows at the Mount Sinai Health System. This role will ensure that these models meet Mount Sinai’s high standards for safety, equity, and real-world performance.
You will work in partnership with the AI Governance Committee, product owners, clinicians, Epic technical team, and DevOps teams to rigorously evaluate both Generative and Non-Generative AI tools before and after deployment.
ResponsibilitiesCore ML & Validation
- Curate, clean, and manage large complex datasets from various sources for modeling and analysis.
- Assist in designing, evaluating, and refining ML models, including performance benchmarking and fairness analysis.
- Develop and run model validation tests for robustness, accuracy, and generalizability across demographic subgroups.
- Collaborate on model QA and deployment workflows with engineering teams, ensuring models are safe and production ready.
AI Product Governance
- Design validation protocols for AI models to ensure ethical use, bias mitigation, and regulatory compliance.
- Monitor deployed models for performance degradation, drift, and bias over time.
- Ensure auditability of all validation and monitoring outputs using standardized documentation practices.
- Assist in compliance workflows including traceability, re-validation, and version documentation of AI tools.
- Ensure products maintain expected performance in clinical settings by tracking model drift, bias, and data integrity issues
Communication & Reporting
- Translate statistical and technical insights into accessible reports for clinicians, product managers, and governance bodies.
- Present results from AI validation testing and monitoring to stakeholders and contribute to institutional governance decisions.
- Maintain rigorous documentation on model performance, validation methods, and compliance actions.
- Stay current with emerging best practices in AI model development, validation, safety, transparency, and interpretability.
Qualifications- Master's degree in a quantitative discipline (e.g., Statistics, Operations Research, Bioinformatics, Economics, Computational Biology, Computer Science, Information Technology, Mathematics, Physics) or equivalent practical experience.
- 2 years of work experience in data science, software engineering, or data analysis
- Experience with at least one programming language among Scala, Python, Java, C, or C++.
- Expert knowledge on Machine Learning Algorithms
- Proficiency in database languages (e.g., SQL, NoSQL) and cloud computing platforms (e.g., AWS, Azure, GCP)
- Proficiency in visualization tools like Plotly, Tableau, Power BI
- Familiarity with ML lifecycle management tools (e.g., MLflow, Kubeflow, Airflow) Experience with monitoring tools for AI model tracking
- Understanding of DevOps principles, CI/CD pipelines, and containerization (e.g., Docker, Kubernetes)
- Experience with version control systems (e.g., Git)
- Knowledge of big data technologies (e.g., Hadoop, Spark)
- Strong problem-solving skills and ability to work in cross-functional teams