Minimum Qualifications:Strong Experience in Spark Streaming/API projects Experience in distributed architecture and big data technologies / systems such as but not limited to the following:
Informatica/Talend
Cloudera
Hadoop Files System (HDFS)
Processing frameworks MapReduce
Spark
Resource Allocator/YARN
Knowledgeable in designing sustainable data architecture for Big data systems. That is balance partitioning with respect to target block s of the HDFS, compaction techniques, bucketing etc.
Experience in managing Small Files in HDFS, Storage formats such as columnar format (Parquet and ORC) and Transmit formats (Avro)
Knowledgeable on handling semi-structured data formats – JSON, XML, HTML etc.
Experienced in using various ETL querying tools for distributed systems such as Hive, Impala and other SQL clients (RDBMS, Phyton, Scala etc,)
Ability to handle and understand data transformation requirements to suit business needs.
Experience in handling tools for data transformation and/or ETL through tool-based and code- based development – ex. Informatica or Talend for tool based / Spark or MapReduce framework in Python or Scala/Java for code-based development.
Knowledgeable in resource allocation and proper resource sizing for distributed jobs. Ex. Spark executor and driver s
Background on data warehousing, master data management and reporting technologies.
Experience in handling streaming technologies ex. Kafka. Understands concepts of Kafka topics, consumer groups and producers and Flume. Required Technical Skills:
Experience in Talend/Informatica, Ab Initio and other ETL GUI based tools
Strong DB and Systems Administration and SQL skills
Experience in any of the following languages Python, Java, Scala
Strong Spark API skills including proper resource allocation and optimizations
Strong Kafka Producer and Consumer API skills
Comfortable in working with Hive partitioning and bucketing Desirable Qualifications:
Experienced in end-to-end DevOps techniques is a plus. Able to create Jenkins pipelines and create unit test cases for automated testing. Experience in Git version control is a plus.
Experience in Docker containers and container technologies is a plus.
Experience in Flink Streaming framework.
Experience in using Cloud Technologies (AWS RedShift, S3 etc.)
Candidate must possess at least a Bachelor's/College Degree, Computer Science/Information Technology, Engineering
At least 2 year(s) of working experience of relevant experience.
