Data Engineer (Python, Pyspark)

Experience: 8+ years

Mandatory Skills: Python, Spark

Pyspark Must

Responsibilities:

• Design, develop, and maintain software solutions using Python .
• Developed Python and pyspark programs for data analysis. Good working experience with python to develop Custom Framework for generating of rules (just like rules engine).
• Developed Python code to gather the data from HBase and designs the solution to implement using Pyspark. Apache Spark DataFrames/RDD's were used to apply business transformations and utilized Hive Context objects to perform read/write operations.
• Develop back-end services, APIs, and integrate databases.
• Optimize applications for performance, security, and maintainability.
• Troubleshoot and resolve software defects and issues.

Required Skills:

Strong proficiency in Python and Spark