Data Engineer

12-24 month

Phone and skype

Onsite – MUST be willing to work On-Site Houston,Texas

Job Title: Data Engineer – Real-time Operations 

Candidate MUST be able to work independently 

Able to code ETL work and show where they have used custom SQL based scripts, not interested in candidates that use Informatica

Solid Python Development skills are necessary

Looking for candidates that ask questions and have worked directly with Business units.

Able to discuss what the data is and why it is being processed, where it is going and what it is intended to do in the way of benefit to end Client.

Job Overview: Our client is looking for talented data engineering professionals who are passionate about leveraging technology to provide meaningful impact through their work. As part of the client's development teams, you will contribute to the development of software solutions that aide in driving efficiencies for drilling, completions, production, and subsurface evaluation activities for a premiere, Fortune 200, Oil & Gas company.

This position and program will require the candidate to be on-site at our client's office in downtown Houston, TX.

Responsibilities:

Data Management & Integration:

· Design and build pipelines to ingest and transport data

· Own data quality and reliability for app-related data pipelines.

· Collaborate with data and product teams to deliver robust, scalable data solutions.

API Development & Management:

· Design and develop secure, robust, user-friendly APIs for applications and users.

· Ensure API performance, fault tolerance, and comprehensive documentation.

Data Storage & Accessibility:

· Build and support databases specific to app usage.

· Work with enterprise database teams to make relevant app data available to stakeholders.

Analytics & Monitoring:

· Generate business rule-based alerts and notifications for stakeholders.

· Develop and support app health monitoring infrastructure to ensure uptime and reliability.

· Enhance and maintain the performance of real-time streaming data processes.

Additional Responsibilities:

· Understand the oil and gas business domain to translate needs into technical solutions.

· Participate in code review processes to ensure code quality and security.

· Provide data quality support and maintenance for assigned applications.

Requirements:

· Minimum 5 years of experience as a Data Engineer or in a similar role.

· Direct exposure to developing scientific or engineering use cases; or degree in a STEM field.

· Strong proficiency in Python preferred.

· Demonstrated experience in stream processing and real-time data calculations, ideally with IoT data.

· Proven ability in working with data caching, relational databases (RDBMS), and API creation/integration.

· Strong problem-solving skills and a proven track record of learning and adapting to new technologies.

· Excellent collaboration and communication skills, with a keen ability to work effectively within dynamic team environments.

Our Approach to Technology:

We prioritize finding the right cultural fit and strong technical aptitude, with a willingness to learn and adapt. While we've listed technologies based on past implementations, we are flexible to similar experience and value candidates who can demonstrate their ability to excel in a dynamic environment.

· Understanding of Relational Database Management Systems (RDBMS)

o Oracle, SingleStore, MySQL, or Postgres

· Experience with NoSQL Databases

o Redis, MongoDB, and Elasticsearch

· Exposure to Event Log/Message Queuing Systems

o RabbitMQ, Pulsar, or Kafka

· API Development and Integration

o API creation concepts and experience with Python and FastAPI is a benefit

· Stream Processing Expertise

o Experience with large-scale IoT stream processing using Python for real-time data calculations and transformations (on data volumes exceeding MBs per second) is desirable.

· Infrastructure and Deployment Skills

o GitHub Actions, Kubernetes, Docker, Terraform, and Azure for deployment and infrastructure management on Linux (CentOS)

Please review the feedback that we got from my business partner regarding a few applicants for the post of data engineer.

• Unfortunately, he has been associated with single Large applications that have many working on the team.   In this regard, he has limited interaction with user groups
• When asked about what he suggested or recommended on the project , the answer was a bit vague , nothing he could point to as something he did.
• Relies on tools like Informatica to handle ETL work and EOG does not use these tools , they use custom SQL scripting
• Has not really done full application role as an individual contributor since he was in India

I just spoke with him,  Nice fellow but he has never been in a situation where he could work directly with both requesting party and users to engineer a solution.  Has always had a large team structure and full PM and BA compliment to support needs he has been given.    Also has not been given any opportunity to make suggestion or recommendations during the assignment at Carnival for almost 4 years!  Uses the tech they give him.  He is a very nice fellow but will not e a fit for the dynamic environment the Client has .  

Client is looking for candidate that has worked independently as a Data Engineer

Someone who is interested in what the job is ;  why the data is needed, who has it, does it need to be transformed into a different format , do we have space on target servers and what is the value of the data to the end Client.They need to be able to talk about their accomplishments not as a collective "We"

They need to show they have had some initiative in making suggestions and recommendations based on their own research in regard to new technologies, algorithms or other techniques to the work they have been assigned.  Last,  I need someone that is both technically sound and curious .