Responsibilities

• Working as part of the ML engineering team, providing the data for model training
• Create and maintain optimal data pipeline architecture
• Building and improving the data acquisition process
• Writing code to help integrate the data pipelines & deploying them to production
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Verifying data quality, and/or ensuring it via data cleaning
• Defining validation strategies, and writing the relevant tests for them
• Defining the preprocessing or feature engineering to be done on a given dataset
• Create data tools for ML team members that assist them in building and optimizing our product
• Build processes supporting data transformation, data structures, metadata, dependency and workload management
• If needed, work closely with the company’s security experts to help translate their knowledge into actionable algorithms and/or data processing methodologies

Required Skills

• Experience building and optimizing ‘big data’ data pipelines, architectures and data sets
• Strong analytic skills related to working with unstructured datasets
• Good grasp of design pattern concepts and data structures 
• Experience with relational SQL databases, primarily Postgres
• Proven background with an OO programming language

Advantages

• Good grasp of the server-client communication process via protocols such as HTTP/S
• Familiarity with Linux
• Working knowledge of Ruby or Crystal languages