Description
Summary Development of multiple Map Reduce jobs in java and through Cascading API for data cleansing and preprocessing. Defining processes for loading data from RDBMS into HDFS using Sqoop. Defining processes for loading data from various sources into HDFS and HBase using Flume. Development of PIG Jobs scripts for data cleansing and data ingestion. Defining and automating the jobs deployment to Hadoop cluster with scheduling Oozie workflow to run the Jobs. Using Ranger for the security and also build the Web application Api for the Operations team to define policy and access for Data-Lake. Development on Spark SQL and Spark Streaming using Scala. Worked on writing shell scripts for validating different checks on data through running hive queries. Worked with gradle for deployment of the Hadoop Jobs through Knox to DataLake. Worked with Jenkins gradle for creation of Jenkins jobs to be used for the deployment of the different workflow/coordinator Hadoop Jobs to DataLake in different environments (Dev/Tst/AT/Prod) using Knox Using code versioning using GIT. Design the application, participate in design discussions, and review design artifacts. Handle client communication regarding requirements, design, etc. Review the developed code and make sure it adheres to the design, standards and guidelines of the clients and Virtusa. Providing the solution to fix and support the priority bugs in TEST, SIT, UAT and Pre-Production environments. Following the Agile methodology like Daily Standup meetings with our Scrum Master, Status call with clients and Defects Triage Meetings. Communicated with onsite coordinator for requirements understanding and clarifications. Reviewing performance and code quality of the application. Designed and developed base classes, framework classes and common re-usable Components. Participate in meetings related to project management (with the client) and related to technical deliveries. Deployment support for minor/major releases. Manage onsite incidents.
Work Experience
| COMPANY | POSITION HELD | DATES WORKED |
|---|---|---|
| (Confidential) | Hadoop Technical Lead | 8/2016 - Present |
Accomplishments
Highlights:
Keywords
