[awslogsencodingextension] Establish a standard format for custom timestamp fields

[MichaelKatsoulis]

Component(s)

extension/encoding/awslogsencoding

Describe the issue you're reporting

Problem

The awslogsencodingextension currently has no established handling of custom timestamp fields that are not the official record timestamp:

1. VPC Flow Logs: aws.vpc.flow.start field uses int64 (Unix seconds since epoch)
2. Future timestamp fields: No standard for how custom timestamp fields should be formatted

The official record timestamps are correctly handled as nanoseconds since epoch (OpenTelemetry standard), but custom timestamp fields like aws.vpc.flow.start need a consistent approach.

Current State

Field	Type	Format	Issue
aws.vpc.flow.start	int64	Seconds since epoch	Inconsistent with semconv patterns
Record timestamps	int64	Nanoseconds since epoch	Correct (OpenTelemetry standard)

Problem Details

• SemConv patterns: Semantic conventions don't have a way to set a field as a date type. In most existing semconv fields ISO-8601 formatted date is used. Few are also in nanoseconds since epoch
• Consistency: Need a standard for future custom timestamp fields
• Consumer Experience: Integer timestamps are not human-readable. They require additional handling from extension consumers

Proposed Solution

Standard for Custom Timestamp Fields:

• Type: string
• Format: ISO-8601 with millisecond precision
• Pattern: "2006-01-02T15:04:05.000Z"
• Timezone: UTC (indicated by Z suffix)

Examples

Current :

{
  "timestamp": 1609459200000000000,           // Record timestamp (nanoseconds) 
  "aws.vpc.flow.start": 1609459200            // Custom field (seconds) 
}

Proposed:

{
  "timestamp": 1609459200000000000,           // Record timestamp (nanoseconds) 
  "aws.vpc.flow.start": "2021-01-01T00:00:00.000Z"  // Custom field (ISO-8601) 
}

Decision

After discussions in the issue we decided to go with RFC3339Nano for nanoseconds precision.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• extension/encoding/awslogsencoding: @axw @constanca-m

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[axw]

I'm on board with a string-formatted timestamp, I like that it's more self-evidently a timestamp. Precedent:

• https://opentelemetry.io/docs/specs/semconv/registry/attributes/process/#process-creation-time
• https://opentelemetry.io/docs/specs/semconv/registry/attributes/process/#process-exit-time
• https://opentelemetry.io/docs/specs/semconv/registry/attributes/tls/#tls-server-not-after
• https://opentelemetry.io/docs/specs/semconv/registry/attributes/tls/#tls-server-not-before

[axw]

@MichaelKatsoulis assigning to you since you've opened #43390

[joecompute]

@axw @MichaelKatsoulis so, GCP has a bit more precision here and provides its start and end data as RFC3339 with nanoseconds. I understand that RFC3339 is a subset ("profile") of ISO8601. For the sake of consistency and preventing information loss, would it be possible to stick with the nanosecond precision via RFC 3339 with nanoseconds? In Go's time package, they have an easy format option for this which is listed across the codebase (time.RFC3339Nano).

Example sampled GCP record

[MichaelKatsoulis]

Thanks @joecompute for looking this up. I would love to hear @ChrsMark take on this standard from the semantic convention point of view.