Splunk Engineer

The proposed Splunk Engineer shall have the following qualifications:

• At least 3 years experience working on Splunk system
• Possess Splunk Enterprise Certified Admin certifications or equivalent.

Working arrangement:

1. To able to work independently and ensure smooth running of the System. Official working hours: 8.30am to 6.00pm (Monday to Thursday), 8.30 to 530pm (Friday) and based on-site.

The Splunk Engineer shall perform critical high-risk works during maintenance windows specified by the Client, which may be off-office hours or during weekends.

The Splunk Engineer shall be responsible of all the corrective and preventive maintenance of the Splunk systems in all environments.

The Splunk Engineer shall remediate all vulnerabilities or penetration test findings pertaining to the Splunk systems.

The Splunk Engineer can raise tickets to Splunk principal for support and queries.  

System Operations

a)       Perform checks and troubleshoot, if necessary, to ensure the Client’s Splunk services are running as intended for all environments.

b)       Maintain and monitor Splunk infrastructure (Search Heads, Indexers, Forwarders, Deployment Server, Cluster Master, etc.).

c)       Ensure uptime and system health via monitoring, tuning, and log analysis (including introspection, metrics logs).

d)       Manage indexing performance and storage usage: data retention, index lifecycle, bucket management. 

e)       Generate and check reports from the system to ensure the system and agents are working as intended

f)        Perform checks and troubleshoot, if necessary, to ensure that the Splunk forwarders (agents) are working and can pipe logs back to Splunk systems.

g)       Perform checks and troubleshoot, if necessary, to ensure the Splunk systems can receive logs from sources such as CloudWatch or syslog servers.

h)       Integrate Splunk with the Client’s systems and processes to perform real-time monitoring and alert when Splunk infrastructure is not working well, so that issues can be attended to early. (e.g., log breaks, disconnected agents, search-head hung from insufficient resources, etc.)

i)         Fine tune Splunk rules according to the Client’s request.

j)         Perform parser validation or write new custom parser according to the Client’s request

k)       Work closely with the Client’s SOC to ensure Splunk supports threat detection, auditing, and incident response use cases.

l)         Change the passwords for all privilege and services accounts for the Splunk systems regularly

m)     Ensure the Splunk systems is working as intended during the Client’s periodic BCP and DR exercises. 

Problem Resolution

a)       Investigate problems and provide assistance to triage issues.

b)       Correct defects in the System, including temporary corrections or workarounds until permanent fixes or updates are available.

c)       Prepare incident report including the root cause analysis and necessary resolution

d)       Track and report issues, support cases and incident resolutions on a weekly basis.

System Monitoring

a)       Monitor Security advisory, new releases, notifications and maintenance expiry dates for all Software used in the System and assess the impact, if any.

b)       Recommend to the Client the best course of action to take and provide all relevant documentation.

c)       If the issue arises from a security vulnerability or software incompatibility, the RE shall evaluate and implement fixes to address the vulnerability or incompatibility.

d)       Check and remediate findings from the Client’s periodic vulnerability and compliance scans.

e)       Track and update the Client on the DLP End of Life (EOL) and End of Support (EOS) and plans to maintain product supportability.

System Changes

a)       Deploy and test system changes in the non-Production environments when required.

b)       Demonstrate that System functionality and performance are not degraded.

c)       Implement the system changes into the Production environment upon the Client’s acceptance of the testing results.

d)       Implementation of additional use cases, report design and development and tuning to reduce false positives and negatives.

Documentation

a)       Create or provide the Client with all System related documentation, including standards and procedures, operation manuals, workflows, processes, etc.

b)       Update the relevant documentation when changes are made to the System or processes.