Principal Observability Architect (Splunk & Databricks)

Scicom Infrastructure Services, Inc. is hiring a Principal Observability Architect to lead the design, implementation, modernization, and optimization of enterprise-scale observability and analytics platforms. This is a senior technical authority role for someone with deep experience in both traditional observability platforms and modern data lakehouse architectures. The position is based in the United States.

What you will do

• Define enterprise observability architecture standards, patterns, and roadmaps.
• Lead observability transformation initiatives involving Splunk modernization and Databricks adoption.
• Develop reference architectures for telemetry ingestion, storage, analytics, security, and AI-driven operations.
• Align observability strategies with business, security, compliance, and operational objectives.
• Create executive-level architecture presentations, business cases, and technology roadmaps.
• Architect large-scale Splunk Enterprise and Splunk Cloud environments, including indexer clusters, search head clusters, forwarder architectures, deployment servers, data models, and ITSI implementations.
• Define ingestion, retention, indexing, and data lifecycle strategies.
• Lead migration initiatives involving Splunk to Databricks, Heavy Forwarders to Cribl, and SIEM modernization programs.
• Optimize SPL searches, data models, summary indexing, and dashboard performance.
• Architect enterprise observability data lake solutions using Databricks Lakehouse, Delta Lake, Unity Catalog, Delta Live Tables, Structured Streaming, Mosaic AI, and Genie.
• Design Medallion Architectures (Bronze, Silver, Gold).
• Develop governance strategies including RBAC, data masking, data lineage, and audit controls.
• Create high-performance log analytics solutions capable of supporting petabyte-scale telemetry environments.
• Enable self-service analytics and AI-powered observability use cases.
• Design ingestion architectures supporting OpenTelemetry, OCSF, Syslog, Kafka, Azure Event Hubs, AWS Kinesis, GCP Pub/Sub, and Cribl.
• Define normalization and enrichment frameworks, and establish data quality and schema management processes.
• Design real-time and batch processing pipelines.
• Lead implementation of AIOps, predictive analytics, root cause analysis, anomaly detection, and event correlation.
• Integrate observability datasets with AI/ML platforms.
• Develop observability use cases leveraging Mosaic AI, Agentic AI, LLMs, and Generative AI.
• Build operational intelligence and executive KPI dashboards.
• Architect observability solutions supporting SOC operations, threat hunting, security analytics, and compliance reporting.
• Design frameworks aligned with HIPAA, PCI-DSS, SOX, NIST, and ISO 27001.
• Implement data governance and security controls across observability platforms.
• Provide technical leadership to engineering teams, mentor architects, engineers, and developers, conduct architecture reviews, and define platform standards.
• Engage directly with executive stakeholders and business leaders.

What they are looking for

• 10+ years of experience in Enterprise Observability, Monitoring, or Security Analytics.
• 5+ years architecting large-scale Splunk environments.
• 3+ years designing Databricks Lakehouse architectures.
• Experience managing environments exceeding 50 TB/day preferred, 100+ TB/day strongly preferred.
• Experience leading enterprise transformation programs.
• Deep expertise in Splunk Enterprise, Splunk Cloud, Splunk ITSI, Enterprise Security, SPL Development, Data Models, Indexer Clustering, Search Head Clustering, SmartStore, Heavy Forwarders, and Universal Forwarders.
• Strong experience with Databricks Lakehouse, Delta Lake, Unity Catalog, Delta Live Tables, Structured Streaming, Databricks SQL, Genie, Mosaic AI, and Lakehouse Federation.
• Experience with one or more cloud platforms: Microsoft Azure, Amazon Web Services, or Google Cloud.
• Strong knowledge of Kafka, OpenTelemetry, OCSF, Iceberg, Spark, SQL, Python, REST APIs, and Event Streaming Architectures.

Preferred qualifications

• Experience with Cribl Stream and Cribl Edge.
• Experience with Dynatrace, AppDynamics, Datadog, or New Relic.
• Experience with ServiceNow ITOM/Event Management.
• Experience designing AI/ML operational analytics solutions.
• Experience with Security Data Lakes and SIEM modernization initiatives.
• Experience with FinOps and cloud cost optimization.
• Experience building observability platforms for healthcare, financial services, retail, or large enterprise organizations.

Certifications (preferred)

• Splunk Enterprise Certified Architect
• Splunk Core Certified Consultant
• Databricks Certified Data Engineer Professional
• Databricks Certified Solutions Architect
• Azure Solutions Architect Expert
• AWS Solutions Architect Professional
• Google Professional Cloud Architect

Success metrics

Within the first 12 months, the architect will deliver an enterprise observability architecture roadmap, reduce observability platform costs through modernization initiatives, design and implement a scalable observability data lake architecture, improve telemetry ingestion performance and reliability, enable AI-powered analytics and operational intelligence capabilities, establish enterprise governance standards for observability and security telemetry, and support petabyte-scale observability and security analytics workloads.

Ideal background

Candidates from organizations utilizing large-scale observability environments such as healthcare, banking, retail, telecommunications, logistics, cloud providers, or managed services organizations are highly desirable. Experience supporting environments generating 100TB+ of telemetry per day and integrating Splunk, Databricks, Cribl, OpenTelemetry, and cloud-native data platforms is strongly preferred.