Senior Data Engineer Big Data & Cloud Integration L3

New Yesterday

Benefits: HYBRID Competitive salary Opportunity for advancement
Senior Data Engineer - Big Data & Cloud Integration DALLAS, TX - HYRBID LONG TERM IN-PERSON INTERVIEW Data Engineer - L3 ROLE
- Translate complex cross-functional business requirements and functional specifications into logical program designs and data solutions. - Partner with the product team to understand business needs and specifications. - Solve complex architecture, design and business problems. - Coordinate, Execute and participate in component integration (CIT) scenarios, system integration testing (SIT), and user acceptance testing (UAT) to identify application errors and to ensure quality software deployment. - Continuously work with cross-functional development teams (Data Analysts and Software Engineers) for creating PySpark jobs using Spark SQL and help them build reports on top of data pipelines. - Build, test and enhance data curation pipelines, integrate data from a wide variety of sources like DBMS, File systems and APIs for various OKRs and metrics development with high data quality and integrity. - Execute the development, maintenance, and enhancements of data ingestion solutions of varying complexity levels across various data sources like DBMS, File systems (structured and unstructured), APIs and Streaming on on-prem and cloud infrastructure. - Responsible for the design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms. - Work with building data warehouse structures, and creating facts, dimensions, aggregate tables, by dimensional modeling, Star and Snowflake schemas. - Develop spark applications in PySpark on distributed environment to load huge number CSV files with different schema in to Hive ORC tables. - Perform ETL transformations on the data loaded into Spark Data Frames and do the in-memory computation. - Develop and implement data pipelines using AWS services such as Kinesis, S3 to process data in real-time. - Work with monitoring, logging and cost management tools that integrate with AWS. - Schedule the spark jobs using Airflow scheduler to monitor their performance.
Flexible work from home options available.
Compensation: $60.00 - $65.00 per hour
About Us
We work to deliver profitability in your business - with effective communication, consulting, and interactive solutions. Following an Agile Work Approach, we make sure you get the ideal solutions at minimum expenses.
Work Approach
Our Philosophy Our Philosophy starts-and-ends at the Client-first approach. Be it understanding your business requirements to choosing the right technologies, we work as a collective team that takes all the possible steps to grow continuously towards our common goal.
Work Policy
We promote a collaborative work environment. We involve everyone working in the organization in community decisions and encourage them to think from a broader perspective. Our work process promotes flexibility and we maintain a high level of discipline at different levels of execution.
The Future
SelectMinds have years of experience in the domain helps us understand the need-of-the-hour better. This understanding drives us to a better future with every minute ticking. We believe we will be taking off major businesses from their flagship positions, with the products we are eyeing today.
Apply
Location:
Dallas

We found some similar jobs based on your search