-
Notifications
You must be signed in to change notification settings - Fork 2
/
draft-ietf-opsawg-ipfix-on-path-telemetry-05.txt
1624 lines (999 loc) · 57.4 KB
/
draft-ietf-opsawg-ipfix-on-path-telemetry-05.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Network Working Group T. Graf
Internet-Draft Swisscom
Intended status: Standards Track B. Claise
Expires: 9 May 2024 Huawei
A. Huang Feng
INSA-Lyon
6 November 2023
Export of On-Path Delay in IPFIX
draft-ietf-opsawg-ipfix-on-path-telemetry-05
Abstract
This document introduces new IP Flow Information Export (IPFIX)
information elements to expose the On-Path Telemetry measured delay
on the IOAM transit and decapsulation nodes.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 9 May 2024.
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Graf, et al. Expires 9 May 2024 [Page 1]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Performance Metrics . . . . . . . . . . . . . . . . . . . . . 6
3.1. IP One-Way Delay Hybrid Type I Passive Performance
Metrics . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.1.1. Summary . . . . . . . . . . . . . . . . . . . . . . . 6
3.1.2. Description . . . . . . . . . . . . . . . . . . . . . 7
3.1.3. Change Controller . . . . . . . . . . . . . . . . . . 7
3.1.4. Version of Registry Format . . . . . . . . . . . . . 8
3.2. Metric Definition . . . . . . . . . . . . . . . . . . . . 8
3.2.1. Reference Definition . . . . . . . . . . . . . . . . 8
3.2.2. Fixed Parameters . . . . . . . . . . . . . . . . . . 8
3.3. Method of Measurement . . . . . . . . . . . . . . . . . . 9
3.3.1. Reference Methods . . . . . . . . . . . . . . . . . . 9
3.3.2. Packet Stream Generation . . . . . . . . . . . . . . 9
3.3.3. Traffic Filtering (Observation) Details . . . . . . . 9
3.3.4. Sampling Distribution . . . . . . . . . . . . . . . . 9
3.3.5. Runtime Parameters and Data Format . . . . . . . . . 9
3.3.6. Roles . . . . . . . . . . . . . . . . . . . . . . . . 10
3.4. Output . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.4.1. Type . . . . . . . . . . . . . . . . . . . . . . . . 10
3.4.2. Reference Definition . . . . . . . . . . . . . . . . 11
3.4.3. Administrative Items . . . . . . . . . . . . . . . . 13
3.4.4. Comments and Remarks . . . . . . . . . . . . . . . . 13
4. IPFIX Information Elements . . . . . . . . . . . . . . . . . 13
5. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15
6.1. Performance Metrics . . . . . . . . . . . . . . . . . . . 15
6.2. IPFIX Entities . . . . . . . . . . . . . . . . . . . . . 15
6.2.1. PathDelayMeanDeltaMicroseconds . . . . . . . . . . . 16
6.2.2. PathDelayMinDeltaMicroseconds . . . . . . . . . . . . 17
6.2.3. PathDelayMaxDeltaMicroseconds . . . . . . . . . . . . 17
6.2.4. PathDelaySumDeltaMicroseconds . . . . . . . . . . . . 17
7. Operational Considerations . . . . . . . . . . . . . . . . . 18
7.1. Time Accuracy . . . . . . . . . . . . . . . . . . . . . . 18
7.2. Mean Delay . . . . . . . . . . . . . . . . . . . . . . . 18
7.3. Reduced-size encoding . . . . . . . . . . . . . . . . . . 18
7.4. IOAM Application . . . . . . . . . . . . . . . . . . . . 18
8. Security Considerations . . . . . . . . . . . . . . . . . . . 19
9. Implementation Status . . . . . . . . . . . . . . . . . . . . 19
9.1. FD.io VPP . . . . . . . . . . . . . . . . . . . . . . . . 19
9.2. Huawei VRP . . . . . . . . . . . . . . . . . . . . . . . 19
9.3. Fluvia . . . . . . . . . . . . . . . . . . . . . . . . . 20
9.4. Pmacct Data Collection . . . . . . . . . . . . . . . . . 20
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 20
Graf, et al. Expires 9 May 2024 [Page 2]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
11.1. Normative References . . . . . . . . . . . . . . . . . . 20
11.2. Informative References . . . . . . . . . . . . . . . . . 20
Appendix A. IPFIX Encoding Examples . . . . . . . . . . . . . . 24
A.1. Aggregated On-Path Dealay Examples . . . . . . . . . . . 24
A.1.1. Template Record and Data Set with Mean Delta . . . . 24
A.1.2. Template Record and Data Set with Sum Delta . . . . . 26
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28
1. Introduction
Network operators want a statistical delay view of their networks.
They want to understand where in the network, for which customer
traffic, how much and why delay is being accummlated. In order to
answer why and where, delay needs to be reported into device and
control-plane context. In order to understand which customer traffic
is affected, delay needs to be reported into customer data-plane
context. That enables network operators to quickly identify when the
control-plane updates the current path with a different next-hop and
therefore the forwarding path changes to different nodes and
interfaces, how the path delay changes for which customer traffic.
With On-Path Telemetry, described in the Network Telemetry Framework
[RFC9232] and applied in In-situ OAM [I-D.ietf-ippm-ioam-deployment],
Path Tracing [I-D.filsfils-spring-path-tracing] and In-situ Flow
Information Telemetry [I-D.song-opsawg-ifit-framework], the path
delay between two endpoints is measured by inserting a timestamp in
the packet.
On-Path Telemetry can be distinguished between two modes. Passport
mode, [RFC9197], where only the last hop in the forwarding path of
the On-Path Telemetry domain exposes all the metrics, and postcard
mode, [I-D.song-ippm-postcard-based-telemetry], where the metrics are
also exposed in the transit nodes. In both modes the forwarding path
exposes performance metrics allowing to determine how much delay has
been accumulated on which hop.
This document defines four new IPFIX Information Elements (IEs),
exposing the On-Path delay on IOAM transit and decapsulation nodes,
following the postcard mode principles. Since these IPFIX IEs are
performance metrics [RFC8911], they must be registered in the "IANA
Performance Metric Registry [IANA-PERF-METRIC].
Following the guidelines for Registered Performance Metric requesters
and reviewers [RFC8911], the different characteristics of the
performance metrics (Identifier, Name, URI, Status, Requester,
Revision, Revision Date, Description, etc) must be clearly specified
in the "IANA Performance Metric Registry [IANA-PERF-METRIC] in order
for the results of measurements using the Performance Metrics to be
Graf, et al. Expires 9 May 2024 [Page 3]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
comparable even if they are performed by different implementations
and in different networks. These characteristics start by selecting
a meaningful name, following the "MetricType_Method_SubTypeMethod_...
Spec_Units_Output" naming convention (See Section 7.1.2 of
[RFC8911]).
+------------------------------------+-------------------------------+
| Performance Metric | IPFIX Information Element |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I |PathDelayMeanDeltaMicroseconds |
|P_RFC[RFC-to-be]_Seconds_Mean (TBD1)|(TBD5) |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I |PathDelayMinDeltaMicroseconds |
|P_RFC[RFC-to-be]_Seconds_Min (TBD2) |(TBD6) |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I |PathDelayMaxDeltaMicroseconds |
|P_RFC[RFC-to-be]_Seconds_Max (TBD3) |(TBD7) |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I |PathDelaySumDeltaMicroseconds |
|P_RFC[RFC-to-be]_Seconds_Sum (TBD4) |(TBD8) |
+------------------------------------+-------------------------------+
Table 1: Correspondance between IPFIX IE and Performance Metric
The delay is measured by calculating the difference between the
timestamp imposed with On-Path Telemetry in the packet at the IOAM
encapsulation node and the timestamp exported in the IPFIX flow
record from the IOAM transit and decapsulation nodes. The lowest,
highest, mean, and/or the sum of measured path delay can be exported,
thanks to the different IPFIX IE specifications.
Graf, et al. Expires 9 May 2024 [Page 4]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
On-Path Telemetry Domain
.........................................
. .
. D1 .
. <------> .
. .
. D2 .
. <--------------------> .
. .
. D3 .
. <-----------------------------------> .
. .
(H1) ------ (R1) ------- (R2) ------- (R3) -------- (R4) ------ (H2)
Host 1 Encapsulation Transit Transit Decapsulation Host 2
Node Node 1 Node 2 Node
. .
. .
.........................................
Figure 1: Delay use case. Packets flow from host 1 to host 2.
On the usecase showed in Figure 1 using On-path Telemetry to export
the delay metrics, the node R2 exports the delay D1, the node R3
exports the delay D2 and the decapsulation node R4 exports the total
delay D3 using IPFIX.
2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
This document makes use of the terms defined in [RFC7011] and
[I-D.ietf-ippm-ioam-deployment].
The following terms are used as defined in [RFC7011].
* IPFIX
* IPFIX Information Elements (IEs)
* Flow Record
* Exporter
The following terms are used as defined in [RFC8911].
Graf, et al. Expires 9 May 2024 [Page 5]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
* Performance Metric
* Registered Performance Metric
* Performance Metrics Registry
The following terms are used as defined in
[I-D.ietf-ippm-ioam-deployment].
* IOAM encapsulation node
* IOAM transit node
* IOAM decapsulation node
3. Performance Metrics
This section defines and describes the new performance metrics by
applying the template defined in Section 11 of [RFC8911].
3.1. IP One-Way Delay Hybrid Type I Passive Performance Metrics
This section specifies four performance metrics for the Hybrid Type I
Passive assessment of IP One-Way Delay, to be registered in the "IANA
Performance Metric Registry [IANA-PERF-METRIC].
All column entries besides the ID, Name, Description, and Output
Reference Method categories are the same; thus, this section defines
four closely related performance metrics. As a result, IANA has
assigned corresponding URLs to each of the four registered
performance metrics.
3.1.1. Summary
This category includes multiple indexes of the registered performance
metrics: the element ID and Metric Name.
3.1.1.1. ID (Identifier)
<insert a numeric Identifier, an integer, TBD>
3.1.1.2. Name
IANA has allocated the numeric Identifiers TBD1-4 for the four Named
Metric Entries in this section
Graf, et al. Expires 9 May 2024 [Page 6]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
3.1.1.3. Name
TBD1: OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Mean
TBD2: OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Min
TBD3: OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Max
TBD4: OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Sum
3.1.1.4. URI
URL: https://www.iana.org/assignments/performance-metrics/
OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Mean
URL: https://www.iana.org/assignments/performance-metrics/
OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Min
URL: https://www.iana.org/assignments/performance-metrics/
OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Max
URL: https://www.iana.org/assignments/performance-metrics/
OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Sum
3.1.2. Description
This metric assesses the one-way delay of IP packets constituting a
single connection between two hosts. We consider the measurement of
one-way delay based on a single Observation Point (OP) [RFC7011]
somewhere in the network. The output is the one-way delay for all
successfully forwarded packets expressed as the <statistic> of their
conditional delay distribution, where <statistic> is one of:
* Mean
* Min
* Max
* Sum
3.1.3. Change Controller
IETF
Graf, et al. Expires 9 May 2024 [Page 7]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
3.1.4. Version of Registry Format
1.0
3.2. Metric Definition
This category includes columns to prompt the entry of all necessary
details related to the metric definition, including the immutable
document reference and values of input factors, called "Fixed
Parameters".
3.2.1. Reference Definition
Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, Ed., "A One-
Way Delay Metric for IP Performance Metrics (IPPM)", STD 81, RFC
7679, DOI 10.17487/RFC7679, January 2016, <https://www.rfc-
editor.org/info/rfc7679>. [RFC7679]
Morton, A. and E. Stephan, "Spatial Composition of Metrics", RFC
6049, DOI 10.17487/RFC6049, January 2011, <https://www.rfc-
editor.org/info/rfc6049>. [RFC6049]
Section 3.4 of [RFC7679] provides the reference definition of the
singleton (single value) one-way delay metric. Section 4.4 of
[RFC7679] provides the reference definition expanded to cover a
multi-value sample. Note that terms such as "singleton" and "sample"
are defined in section 2 of [RFC2330].
With the OP [RFC7011] typically located between the hosts
participating in the IP connection, the one-way delay metric requires
one individual measurement between the OP and sourcing host, such
that the Spatial Composition [RFC6049] of the measurements yields a
one-way delay singleton.
3.2.2. Fixed Parameters
Traffic Filters:
IPv4 header values:
DSCP: Set to 0
IPv6 header values:
DSCP: Set to 0
Hop Count: Set to 255
Flow Label: Set to 0
Extension Headers: None
Graf, et al. Expires 9 May 2024 [Page 8]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
3.3. Method of Measurement
This category includes columns for references to relevant sections of
the RFC(s) and any supplemental information needed to ensure an
unambiguous method for implementations.
3.3.1. Reference Methods
The foundational methodology for this metric is defined in section 4
of [RFC7323] using the Timestamps option with modifications that
allow application at a mid-path OP [RFC7011].
3.3.2. Packet Stream Generation
N/A
3.3.3. Traffic Filtering (Observation) Details
The Fixed Parameters above give a portion of the Traffic Filter.
Other aspects will be supplied as Runtime Parameters (below).
3.3.4. Sampling Distribution
This metric requires a partial sample of all packets that qualify
according to the Traffic Filter criteria.
3.3.5. Runtime Parameters and Data Format
Runtime Parameters are input factors that must be determined,
configured into the measurement system, and reported with the results
for the context to be complete.
The hybrid type I metering parameters must must be reported to
provide the complete measurement context. As an example, if the
IPFIX metering process is used, then the IPFIX metering process
parameters (IPFIX template record used, potential traffic filters,
and potential sampling method and parameters) that generates the flow
records must be reported to provide the complete measurement context.
Src: The IP address of the host in the host A Role (format
ipv4-address-no-zone value for IPv4 or ipv6-address-no-zone value
for IPv6; see section 4 of [RFC6991].
Dst: The IP address of the host in the host B Role (format
ipv4-address-no-zone value for IPv4 or ipv6-address-no-zone value
for IPv6; see section 4 of [RFC6991].
TTL or Hop Limit: Set at desired value.
Graf, et al. Expires 9 May 2024 [Page 9]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
DSCP: Set at desired value.
IPv6 Flow Label: Set at desired value.
Timestamp: The timestamp when the packet is being received at IOAM
encapsulation node. Format depends on On-Path Telemetry
implementation. For IOAM, Section 4.4.1 of [RFC9197] describes
what kind of timestamps are supported. Section 4.4.2.3 and
4.4.2.4 describe where the timestamp is being inserted. For Path
Tracing, Section 4.1 of [I-D.filsfils-spring-path-tracing]
describes what kind of timestamps are supported. Section 9.2
describe the SRH path tracing TLV where the timestamp is being
inserted.
3.3.6. Roles
host A: Launches the IP packet to open the connection. The Role of
"host A" is synonymous with the IP address used at host A.
host B: Receives the IP packet to open the connection. The Role of
"host B" is synonymous with the IP address used at host B.
Encapsulation Node: Receives the IP packet to open the connection
and encapsulates the timestamp into the packet. The Role of
"Encapsulation Node" is synonymous with the timestamp inserted in
the packet.
Transit Node: Receives the IP packet to open the connection and
measures the delay between the timestamp in the packet and the
timestamp when the packet was received.
Decapsulation Node: Receives the IP packet to open the connection
and measures the delay between the timestamp in the packet and the
timestamp when the packet was received and removes the IOAM header
from the packet.
3.4. Output
This category specifies all details of the output of measurements
using the metric.
3.4.1. Type
OWDelay Types are discussed in the subsections below.
Graf, et al. Expires 9 May 2024 [Page 10]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
3.4.2. Reference Definition
For all output types:
OWDelay_HybridType1_Passive_IP: The one-trip delay of one IP packet
is a Singleton
For each <statistic> Singleton one of the following subsections
applies.
3.4.2.1. Mean
The mean SHALL be calculated using the conditional distribution of
all packets with a finite value of one-way delay (undefined delays
are excluded) -- a single value, as follows:
See section 4.1 of [RFC3393] for details on the conditional
distribution to exclude undefined values of delay, and see section 5
of [RFC6703] for background on this analysis choice.
See section 4.2.2 of [RFC6049] for details on calculating this
statistic; see also section 4.2.3 of [RFC6049].
Mean: The time value of the result is expressed in units of seconds,
as a positive value of type decimal64 with fraction digits = 9
(see section 9.3 of [RFC6020]) with a resolution of
0.000000001 seconds (1.0 ns), and with lossless conversion to/from
the 64-bit NTP timestamp as per section 6 of [RFC5905].
3.4.2.2. Min
The minimum SHALL be calculated using the conditional distribution of
all packets with a finite value of one-way delay (undefined delays
are excluded) -- a single value, as follows:
See section 4.1 of [RFC3393] for details on the conditional
distribution to exclude undefined values of delay, and see section 5
of [RFC6703] for background on this analysis choice.
See section 4.3.2 of [RFC6049] for details on calculating this
statistic; see also section 4.3.3 of [RFC6049].
Min: The time value of the result is expressed in units of seconds,
as a positive value of type decimal64 with fraction digits = 9
(see section 9.3 of [RFC6020]) with a resolution of
0.000000001 seconds (1.0 ns), and with lossless conversion to/from
the 64-bit NTP timestamp as per section 6 of [RFC5905].
Graf, et al. Expires 9 May 2024 [Page 11]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
3.4.2.3. Max
The maximum SHALL be calculated using the conditional distribution of
all packets with a finite value of one-way delay (undefined delays
are excluded) -- a single value, as follows:
See section 4.1 of [RFC3393] for details on the conditional
distribution to exclude undefined values of delay, and see section 5
of [RFC6703] for background on this analysis choice.
See section 4.3.2 of [RFC6049] for a closely related method for
calculating this statistic; see also section 4.3.3 of [RFC6049]. The
formula is as follows:
Max = (FiniteDelay[j])
such that for some index, j, where 1 <= j <= N
FiniteDelay[j] >= FiniteDelay[n] for all n
where all packets n = 1 through N have finite singleton delays.
Max: The time value of the result is expressed in units of seconds,
as a positive value of type decimal64 with fraction digits = 9
(see section 9.3 of [RFC6020]) with a resolution of
0.000000001 seconds (1.0 ns), and with lossless conversion to/from
the 64-bit NTP timestamp as per section 6 of [RFC5905].
3.4.2.4. Sum
The sum SHALL be calculated using the conditional distribution of all
packets with a finite value of one-way delay (undefined delays are
excluded) -- a single value, as follows:
See section 4.1 of [RFC3393] for details on the conditional
distribution to exclude undefined values of delay, and see section 5
of [RFC6703] for background on this analysis choice.
See section 4.3.5 of [RFC6049] for details on calculating this
statistic. However in this case FiniteDelay or MaxDelay MAY be used.
Sum: The time value of the result is expressed in units of seconds,
as a positive value of type decimal64 with fraction digits = 9
(see section 9.3 of [RFC6020]) with a resolution of
0.000000001 seconds (1.0 ns), and with lossless conversion to/from
the 64-bit NTP timestamp as per section 6 of [RFC5905].
Graf, et al. Expires 9 May 2024 [Page 12]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
3.4.2.5. Metric Units
The <statistic> of one-way delay is expressed in seconds, where
<statistic> is one of:
* Mean
* Min
* Max
* Sum
The one-way delay of the IP connection singleton is expressed in
seconds.
3.4.2.6. Calibration
Passive Measurements at an OP could be calibrated against an Active
Measurement at host A where the Active Measurement represents the
ground truth.
3.4.3. Administrative Items
3.4.3.1. Status
Current
3.4.3.2. Requester
This RFC
3.4.3.3. Revision
1.0
3.4.3.4. Revision Date
RFC Date
3.4.4. Comments and Remarks
none
4. IPFIX Information Elements
This section defines and describes the new IPFIX IEs.
Graf, et al. Expires 9 May 2024 [Page 13]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
PathDelayMeanDeltaMicroseconds
32-bit unsigned integer that identifies the mean path delay in
microseconds, between the IOAM encapsulation node and the local
node with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
PathDelayMinDeltaMicroseconds
32-bit unsigned integer that identifies the lowest path delay in
microseconds, between the IOAM encapsulation node and the local
node with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
PathDelayMaxDeltaMicroseconds
32-bit unsigned integer that identifies the highest path delay in
microseconds, between the IOAM encapsulation node and the local
node with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
PathDelaySumDeltaMicroseconds
64-bit unsigned integer that identifies the sum of the path delay
in microseconds, between the IOAM encapsulation node and the local
node with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
5. Use Cases
The measured On-Path delay can be aggregated with Flow Aggregation as
defined in [RFC7015] to the following device and control-plane
dimensions to determine:
* With node id and egressInterface(IE14), on which node which
logical egress interfaces have been contributing to how much
delay.
* With node id and egressPhysicalInterface(253), on which node which
physical egress interfaces have been contributing to how much
delay.
* With ipNextHopIPv4Address(IE15) or ipNextHopIPv6Address(IE62), the
forwarding path to which next-hop IP contributed to how much
delay.
* With mplsTopLabelIPv4Address(IE47) or destinationIPv6Address and
srhActiveSegmentIPv6 from [I-D.tgraf-opsawg-ipfix-srv6-srh], the
forwarding path to which MPLS top label IPv4 address or IPv6
destination address and SRv6 active segment contributed to how
much delay.
Graf, et al. Expires 9 May 2024 [Page 14]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
* BGP communities are often used for setting a path priority or
service selection. With bgpDestinationExtendedCommunityList(488)
or bgpDestinationCommunityList(485) or
bgpDestinationLargeCommunityList(491) which group of prefixes
accumulated at which node how much delay.
* With destinationIPv4Address(13), destinationTransportPort(11),
protocolIdentifier (4) and sourceIPv4Address(IE8), the forwarding
path delay on each node from each IPv4 source address to a
specific application in the network.
Taking figure 1 from section 1 as topology example. Below example
table shows the aggregated delay per each node, ingressInterface,
egressInterface, destinationIPv6Address and srhActiveSegmentIPv6.
+-----------+-----------+------+-------------+-------------+------------+
| ingress | egress | Node | destination | srhActive | Path Delay |
| Interface | Interface | | IPv6Address | SegmentIPv6 | |
+-----------+-----------+------+-------------+-------------+------------+
| 271 | 276 | R1 | 2001:db8::2 | 2001:db8::4 | 0 us |
+-----------+-----------+------+-------------+-------------+------------+
| 301 | 312 | R2 | 2001:db8::3 | 2001:db8::4 | 22 us |
+-----------+-----------+------+-------------+-------------+------------+
| 22 | 27 | R3 | 2001:db8::4 | 2001:db8::4 | 42 us |
+-----------+-----------+------+-------------+-------------+------------+
| 852 | 854 | R4 | 2001:db8::4 | 2001:db8::4 | 122 us |
+-----------+-----------+------+-------------+-------------+------------+
Table 2: Example table of measured delay. Ascending by delay.
6. IANA Considerations
6.1. Performance Metrics
This document requests IANA to create new performance metrics under
the "Performance Metrics" registry [RFC8911] with the values defined
in section 2.
6.2. IPFIX Entities
This document requests IANA to create new IPFIX IEs (see table 3)
under the "IPFIX Information Elements" registry [RFC7012] available
at "IANA Performance Metric Registry [IANA-PERF-METRIC] and assign
the following initial code points.
Graf, et al. Expires 9 May 2024 [Page 15]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
+-------+--------------------------------+
|Element| Name |
| ID | |
+-------+--------------------------------+
| TBD5 | PathDelayMeanDeltaMicroseconds |
| | |
+-------+--------------------------------+
| TBD6 | PathDelayMinDeltaMicroseconds |
| | |
+-------+--------------------------------+
| TBD7 | PathDelayMaxDeltaMicroseconds |
| | |
+-------+--------------------------------+
| TBD8 | PathDelaySumDeltaMicroseconds |
| | |
+-------+--------------------------------+
Table 3: Creates IPFIX IEs in the "IPFIX Information Elements" registry
Note to the RFC-Editor:
* Please replace TBD5 - TBD8 with the values allocated by IANA
* Please replace the [RFC-to-be] with the RFC number assigned to
this document
6.2.1. PathDelayMeanDeltaMicroseconds
Name: PathDelayMeanDeltaMicroseconds
ElementID: TBD5
Description: This Information Element identifies the mean path delay
between the IOAM encapsulation node and the local node with the
IOAM domain (either an IOAM transit node or an IOAM decapsulation
node) in microseconds, according to
OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Mean in the
IANA Performance Metric Registry
Abstract Data Type: unsigned32
Data Type Semantics: deltaCounter
Reference: [RFC-to-be], OWDelay_HybridType1_Passive_IP_RFC[RFC-to-
be]_Seconds_Mean in the IANA Performance Metric Registry.
Graf, et al. Expires 9 May 2024 [Page 16]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
6.2.2. PathDelayMinDeltaMicroseconds
Name: PathDelayMinDeltaMicroseconds
ElementID: TBD6
Description: This Information Element identifies the lowest path
delay between the IOAM encapsulation node and the local node with
the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node) in microseconds, according to the
OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Min in the
IANA Performance Metric Registry.
Abstract Data Type: unsigned32
Data Type Semantics: deltaCounter
Reference: [RFC-to-be], OWDelay_HybridType1_Passive_IP_RFC[RFC-to-
be]_Seconds_Min in the IANA Performance Metric Registry.
6.2.3. PathDelayMaxDeltaMicroseconds
Name: PathDelayMaxDeltaMicroseconds
ElementID: TBD7
Description: This Information Element identifies the highest path
delay between the IOAM encapsulation node and the local node with
the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node) in microseconds, according to
OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Max in the
IANA Performance Metric Registry.
Abstract Data Type: unsigned32
Data Type Semantics: deltaCounter
Reference: [RFC-to-be], OWDelay_HybridType1_Passive_IP_RFC[RFC-to-
be]_Seconds_Max in the IANA Performance Metric Registry.
6.2.4. PathDelaySumDeltaMicroseconds
Name: PathDelaySumDeltaMicroseconds
ElementID: TBD8
Description: This Information Element identifies the sum of the path
Graf, et al. Expires 9 May 2024 [Page 17]
Internet-Draft Export of On-Path Delay in IPFIX November 2023
delay between the IOAM encapsulation node and the local node with
the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node) in microseconds, according to
OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Sum in the
IANA Performance Metric Registry.
Abstract Data Type: unsigned64
Data Type Semantics: deltaCounter
Reference: [RFC-to-be], OWDelay_HybridType1_Passive_IP_RFC[RFC-to-
be]_Seconds_Sum in the IANA Performance Metric Registry.
7. Operational Considerations
7.1. Time Accuracy
The same recommendation as defined in section 4.5 of [RFC5153] for
IPFIX applies in terms of clock precision to this document as well.
7.2. Mean Delay
The mean (average) path delay can be calculated by dividing the
PathDelaySumDeltaMicroseconds(TBD5) by the packetDeltaCount(2) at the
IPFIX data collection in order to offload the IPFIX Exporter from
calculating the mean for every Flow at export time.
7.3. Reduced-size encoding
Unsigned64 has been chosen as type for PathDelaySumDeltaMicroseconds
to support cases with large delay numbers and where many packets are
being accounted. As an example, a specific flow record with path
delay of 100 microseconds can not observe more than 42949 packets
without overflowing the unsigned32 counter. The procedure described
in Section 6.2 of [RFC7011] could be applied to reduce network
bandwidth between the IPFIX Exporter and Collector if unsigned32
would be large enough without wrapping around.
7.4. IOAM Application
This document is applicable in IOAM to the Edge-to-Edge and Direct
Exporting Option-Type.
In case of Edge-to-Edge Option-Type, as described in Section 4.6 of