Data Engineer at PETADATA

• Job Location Albany, NY
• Seniority Mid-Senior Level
• Type Full-time

Share on:

Description

Position: Data Engineer Location: Albany, NY (Onsite/Remote) Experience: 13+ Years Work type: C2C only Visa Type: H1-B, GC and US Citizen PETADATA is currently hiring for a Data Engineer for one of their clients.

Roles & Responsibilities Data Pipeline Development Design, build, and maintain scalable data pipelines for ingesting, processing, and transforming data Develop batch and real-time data processing workflows Ensure data pipelines are reliable, efficient, and fault-tolerant Data Architecture & Modeling Design and implement data architectures(data lakes, data warehouses, lakehouses) Create and maintain data models, schemas, and metadata Optimize data storage and retrieval strategies Data Integration Integrate data from multiple sources (databases, APIs, streaming platforms, third-party tools) Handle structured, semi-structured, and unstructured data Implement data validation and quality checks Big Data & Processing Frameworks Work with big data tools such as Apache Spark, Hadoop, Kafka, Flink Build streaming and real-time data processing systems Optimize large-scale data processing performance Cloud & Platform Services Use cloud platforms such as AWS, Azure, or GCP Work with services like: AWS: S3, Glue, Redshift, EMR, Athena Azure: Data Factory, Synapse GCP: BigQuery, Dataflow Manage cloud-based data infrastructure Database Management Manage and optimize SQL and NoSQL databases Implement indexing, partitioning, and performance tuning Support data access for analytics, AI, and reporting teams Data Quality, Security & Governance Implement data quality checks, monitoring, and alerts Enforce data governance, lineage, and metadata management Ensure data security, privacy, and compliance (GDPR, HIPAA, etc.) Automation & Orchestration Automate data workflows using tools like Airflow, Luigi, Prefect Schedule, monitor, and troubleshoot data jobs Reduce manual processes through automation Collaboration & Support Collaborate with data scientists, analysts, and AI engineers Support analytics, BI, and machine learning initiatives Translate business requirements into data solutions Monitoring & Optimization Monitor pipeline performance and data freshness Identify and resolve data issues and bottlenecks Optimize costs and resource usage Optional / Senior-Level Responsibilities Define data engineering standards and best practices Lead data platform design and modernization efforts Mentor junior data engineers Drive data strategy and roadmap planning Required Qualifications Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related field Proven experience as a Data Engineer or similar role Strong programming skills in Python and SQL (Scala or Java preferred) Hands-on experience with big data frameworks such as Spark, Kafka, Hadoop Experience designing and maintaining data pipelines and ETL/ELT processes Strong understanding of data modeling, data warehousing, and lakehouse architectures Experience with cloud platforms: AWS, Azure, or GCP Hands-on experience with SQL and NoSQL databases Experience with workflow orchestration tools (Airflow, Prefect, Luigi) Knowledge of data quality, governance, and security best practices Strong analytical, troubleshooting, and problem-solving skills Excellent communication and collaboration abilities Key Skills & Tools Languages: Python, SQL, Scala, Java Big Data: Spark, Kafka, Hadoop Databases: PostgreSQL, MySQL, MongoDB, Cassandra Cloud: AWS, Azure, GCP Orchestration: Airflow, Prefect Data Warehouses: Redshift, Snowflake, BigQuery Educational Qualification Bachelor’s or Master’s degree in Computer Science or related field We offer a professional work environment and are given every opportunity to grow in the Information technology world.

Note Candidates are required to attend Phone/Video Call / In-person interviews and after Selection of candidate (He/She) should go through all background checks on Education and Experience.

Please email your resume to: [email protected] After carefully reviewing your experience and skills one of our HR team members will contact you on the next steps.