Log Management Requirements

This guide explains the prerequisites for ingesting log data into Conflux. Review supported source types, provider-specific setup requirements, and the network, scale, and retention considerations needed for a reliable deployment.

Supported Log Sources

Conflux can collect:

• Cloud provider logs — API/audit, network, and access logs from AWS, Microsoft Azure, and Google Cloud.
• Host‑level logs — host telemetry and event logs collected through agents such as Metricbeat, Filebeat, Winlogbeat, and Packetbeat.
• Identity & access logs — audit, authentication, and access events from Okta, OneLogin, and other IAM providers.
• Network and security services — events from Cloudflare, WAFs, and other appliances through API, syslog, or cloud storage integrations.
• Custom logs — application logs sent through syslog, HTTPS, external APIs, or file collectors.

Cloud Provider Prerequisites

Review provider‑specific permissions below.

AWS Log Auto-Discovery

For Cloudaware to collect cloud-native logs from AWS:

• Enable CloudTrail, VPC Flow Logs, and other relevant logging services in your accounts.
• Grant Cloudaware permissions so Conflux can:
  • Discover CloudWatch log groups and read events.
  • Access S3 buckets that store logs such as ELB/ALB, CloudFront, and RDS.
• Tag log destinations where tag‑based discovery is supported (for example, log-source:apigateway on CloudWatch log groups, or tags on S3 buckets that store specific logs).

AWS Log Indexes

Log Index	Instruction
log-aws-apigateway	CloudWatch log groups are discovered automatically. Grant logs:DescribeLogGroups, logs:DescribeLogStreams, and logs:GetLogEvents; for tag-based discovery, also grant logs:ListTagsForResource and tag groups with log-source:apigateway.
log-aws-cloudfront-*	Enable CloudFront logging and store logs in an S3 bucket. Grant S3 bucket access with s3:GetObject and s3:ListObject.
log-aws-cloudtrail-*	Enable CloudTrail and ensure the CloudTrail S3 bucket is visible and accessible to Cloudaware.
log-aws-config-*	Enable AWS Config. Grant config:Des*, or at minimum config:DescribeDeliveryChannels.
log-aws-ec2-*	Store EC2 logs in an S3 bucket. Grant S3 bucket access with s3:GetObject and s3:ListObject.
log-aws-eks-*	Enable Amazon EKS logging. Grant logs:DescribeLogGroups, logs:DescribeLogStreams, and logs:GetLogEvents.
log-aws-elb-*	Enable ALB/ELB logging and store logs in an S3 bucket. Grant S3 bucket access with s3:GetObject and s3:ListObject.
log-aws-health-*	Ensure AWS Health events are accessible. Grant health:DescribeEvents, health:DescribeAffectedEntities, and health:DescribeEventDetails.
log-aws-guardduty-*	Store GuardDuty logs in an S3 bucket. Grant S3 bucket access with s3:GetObject and s3:ListObject; if logs are not discovered automatically, tag the bucket with log-source:guardduty.
log-aws-rds-*	Cloudaware tracks RDS logs from CloudWatch and DB instance log events. Use the Cloudaware Conflux Collector policy, which includes the required CloudWatch Logs and RDS log permissions.
log-aws-lambda-*	Cloudaware automatically discovers CloudWatch groups where Lambda logs are stored. Grant logs:DescribeLogGroups, logs:DescribeLogStreams, logs:ListTagsForResource, and logs:GetLogEvents; if search does not return results, tag groups with log-source:lambda.
log-aws-r53resolver-*	Enable Route 53 Resolver DNS query logging.
log-aws-trustedadvisor-*	Ensure Trusted Advisor check data is accessible. Grant support:DescribeTrustedAdvisorChecks and support:DescribeTrustedAdvisorCheckResult.
log-aws-s3accesslog-*	Enable S3 access logging.
log-aws-vpcflowlogs-*	Log VPC, subnet, or Elastic Network Interface traffic to CloudWatch Logs. The destination must be CloudWatch Logs.
log-aws-waf-*	Store WAF logs in an S3 bucket or CloudWatch Logs. Grant stream/S3 discovery permissions (kinesis:DescribeStream, kinesis:ListStreams, s3:ListBucket, s3:GetObject) or CloudWatch Logs permissions; for encrypted S3 buckets, grant kms:Decrypt and kms:DescribeKey.

Refer to the Cloudaware documentation on AWS IAM policies, e.g., CloudTrail, Conflux, and Custom Policy for S3 Decryption (if the S3 bucket with logs is encrypted).

AWS Tag-Based Log Discovery

In environments where standard AWS API discovery is restricted, or where logs are stored in custom S3 prefixes, Conflux supports tag-based discovery as an explicit source-routing method.

To route logs to Conflux, apply the log-source tag key to the relevant Amazon S3 bucket or CloudWatch Log Group. The tag value identifies the log source type, so Conflux can associate the destination with the correct log index and CMDB context.

log-source Tag Values

Apply these values to the log-source tag based on the tagged resource type.

S3 buckets

Tag value	Log source
cloudtrail	AWS CloudTrail logs
elb or alb	Elastic Load Balancer/Application Load Balancer access logs
s3access	S3 server access logs
config	AWS Config logs
cloudfront	AWS CloudFront access logs
vpcflowlogs	VPC Flow Logs
waf	AWS WAF logs
gitlab	GitLab audit/activity logs
guardduty	GuardDuty S3 delivery
trendmicro	Trend Micro Deep Security logs
bedrock	Amazon Bedrock invocation logs
tines	Tines audit logs

CloudWatch Log Groups

Tag value	Log source
rds	Amazon RDS logs
eks	Amazon EKS audit logs
r53	Route 53 DNS logs
waf	AWS WAF CloudWatch logs
ec2	EC2 instance OS logs
lambda	AWS Lambda logs
apigateway	API Gateway logs
bedrock	Amazon Bedrock CloudWatch logs
vpcflowlogs	VPC Flow Logs routed to CloudWatch

Microsoft Azure Log Auto-Discovery

For Cloudaware to collect cloud-native logs from Microsoft Azure:

• Enable diagnostic settings and flow logs on the services you want to monitor.
• Assign Cloudaware the recommended Reader or custom roles so it can read Storage Account keys and log files.
• Ensure export targets (for example, storage accounts) are reachable and sized for expected volume.

Microsoft Azure Log Indexes

Log Index	Instruction
log-azure-activity-*	Assign the Azure Reader role to Cloudaware.
log-azure-flowlogs-*	Create a custom role with Microsoft.Storage/storageAccounts/listKeys/action.

Google Cloud Log Auto-Discovery

For Cloudaware to collect cloud-native logs from Google Cloud:

• Enable audit and other relevant logs in Cloud Logging for the projects you want monitored.
• Configure exports or access patterns that allow Cloudaware to read those logs (for example, in a bucket).

Google Cloud Log Indexes

Log Index	Instruction
log-gcp-audit-*	Enable Google Cloud Logging. Follow Google Cloud Audit Logs documentation.

Host‑Level and On‑Prem Log Discovery

To collect host-level telemetry through agents such as Metricbeat, Filebeat, Winlogbeat, and Packetbeat:

• Ensure outbound connectivity from those hosts to the Conflux endpoint. Ports 8443 and 443 are commonly used, but you should confirm the endpoint and port details for your environment.
• Plan for increased log volume, because host-level agents can produce large amounts of data.

Host‑Level Log Indexes

Log Index	Instruction
metricbeat	Ensure Breeze is installed on a host. Ensure the outbound connection to port 8443 is open on your Conflux node. Warning: once enabled, metricbeat may generate many logs.
winlogbeat	Ensure Breeze is installed on a host. Ensure the outbound connection to port 8443 is open on your Conflux node. Warning: once enabled, winlogbeat may generate many logs.
filebeat	Ensure Breeze is installed on a host. Ensure the outbound connection to port 8443 is open on your Conflux node. Warning: once enabled, filebeat may generate many logs.
packetbeat	Ensure Breeze is installed on a host. Ensure the outbound connection to port 8443 is open on your Conflux node. Warning: once enabled, packetbeat may generate many logs.

For on‑prem network and security devices, you can typically use syslog or HTTPS/APIs to send events to Conflux. Cloudaware will provide the destination hostname, port, and any required authentication tokens.

Third‑Party Apps and Identity Providers Log Discovery

For identity and SaaS sources such as Cloudflare, GitLab audit logs, Okta, and OneLogin:

• Create API tokens with read-only or audit-log permissions, following the provider’s best practices.
• Configure event streaming or exports to the Conflux destinations provided by Cloudaware, such as syslog listener URLs or HTTPS endpoints.
• Where applicable, tag or label the storage locations that contain these logs so CMDB-based discovery can associate them with the correct resources.

Third-Party & IdP Log Indexes

Source	Log Index	Instruction
Cloudflare	log-cloudflare-	Ensure that the API token has read permissions configured at the account level in Cloudflare for Access: Audit Logs and Account Settings.
GitLab Audit	log-gitlab-	Stream: Contact Cloudaware to request a Destination URL for event stream creation in the GitLab account. AWS S3 bucket: Ensure logs are stored in an S3 bucket. Grant Cloudaware access with s3:GetObject and s3:ListObject permissions. Use the log-source => gitlab bucket tag.
Okta	log-okta-system-	Provide Cloudaware support with the Okta URL and a token.
OneLogin	log-onelogin-	Contact Cloudaware to request a Listener URL and token required to create a webhook in OneLogin. Use the provided parameters in Custom headers in the OneLogin UI under Developers > Webhooks. Specify the format as JSON Array.
Syslog	log-syslog-*	Contact Cloudaware to request a syslog server URL. Set up the logs stream to Host: COMPANYNAME-syslog-conflux.cloudaware.com, Port: 514, Protocol: TCP. Replace COMPANYNAME with your organization name.

Network, Scale, and Retention

• Ensure connectivity from cloud and on‑prem sources to Conflux ingestion endpoints (e.g., open egress on required ports and protocols).
• Size storage and retention based on your expected daily log volume and preferred hot vs. archive windows.
• Confirm that your organization’s data governance and retention policies are compatible with Conflux defaults and options.

If you are unsure where to start, begin with:

1. Cloud audit and access logs in one or two key accounts/projects.
2. A small set of high‑value host or application sources.
3. A retention policy that balances investigation needs with cost, then iteratively expand.