1
00:00:00,000 --> 00:00:00,998

2
00:00:00,998 --> 00:00:04,491
[AUDIO LOGO]

3
00:00:04,491 --> 00:00:11,980

4
00:00:11,980 --> 00:00:14,950
Next up is trouble ticket two.

5
00:00:14,950 --> 00:00:18,490
And this one is also
using BGP Topology 1.

6
00:00:18,490 --> 00:00:20,980
Again, we probably won't
point that out again.

7
00:00:20,980 --> 00:00:23,840
But let's go ahead and take
a look at our ticket here.

8
00:00:23,840 --> 00:00:27,850
So it says that router 3
is supposed to be neighbors

9
00:00:27,850 --> 00:00:31,630
with router 7, but it's down.

10
00:00:31,630 --> 00:00:34,480
These are directly
connected-- again, always

11
00:00:34,480 --> 00:00:36,500
starting with
verifying the ticket.

12
00:00:36,500 --> 00:00:38,920
So why don't we go ahead and
jump into the command line

13
00:00:38,920 --> 00:00:41,110
and start on router 3?

14
00:00:41,110 --> 00:00:44,510
So let's start with a
show IP BGP summary.

15
00:00:44,510 --> 00:00:49,960
And we see here that neighbor
7, 10.7.7.7, is currently

16
00:00:49,960 --> 00:00:52,600
in the state OpenConfirm.

17
00:00:52,600 --> 00:00:53,950
So that's interesting.

18
00:00:53,950 --> 00:00:57,850
Let's go look at router
7 and see what it says.

19
00:00:57,850 --> 00:01:00,550
If we look at it
on this side, it's

20
00:01:00,550 --> 00:01:05,910
listing neighbor
10.3.3.3 as OpenSent.

21
00:01:05,910 --> 00:01:08,910
Now before we get too far
into our troubleshooting,

22
00:01:08,910 --> 00:01:12,120
let's talk about some
things that we know already

23
00:01:12,120 --> 00:01:14,650
just from these two outputs.

24
00:01:14,650 --> 00:01:16,993
Well, the obvious is that
the ticket is correct,

25
00:01:16,993 --> 00:01:18,160
and the neighbors aren't up.

26
00:01:18,160 --> 00:01:19,500
OK, fine.

27
00:01:19,500 --> 00:01:23,070
But beyond that, we see
that they're both configured

28
00:01:23,070 --> 00:01:27,390
to peer between their
loopbacks because they're

29
00:01:27,390 --> 00:01:30,360
listing the loopbacks on
both sides as the neighbor's

30
00:01:30,360 --> 00:01:31,240
address.

31
00:01:31,240 --> 00:01:34,920
So they're both properly
configured between loopbacks,

32
00:01:34,920 --> 00:01:38,220
and they can talk to
each other, and there's

33
00:01:38,220 --> 00:01:40,300
nothing like an ACL.

34
00:01:40,300 --> 00:01:44,310
As we saw in ticket one, if
they can't talk to each other

35
00:01:44,310 --> 00:01:47,280
because of either
routing issues or an ACL,

36
00:01:47,280 --> 00:01:49,350
well, then they're
going to be flapping

37
00:01:49,350 --> 00:01:52,920
between active and idle.

38
00:01:52,920 --> 00:01:54,210
These aren't doing that.

39
00:01:54,210 --> 00:01:58,450
They're all the way to
OpenSent and OpenConfirm.

40
00:01:58,450 --> 00:02:01,140
So that lets us know
that they're actually

41
00:02:01,140 --> 00:02:02,770
communicating properly.

42
00:02:02,770 --> 00:02:06,330
So now, let's dig in and start
looking at some other issues.

43
00:02:06,330 --> 00:02:11,460
Since we're here on router 7,
let's do a show IP BGP neighbor

44
00:02:11,460 --> 00:02:13,410
for the 10.3.3.3.

45
00:02:13,410 --> 00:02:17,130
See if we can get some more
detail on this neighborship.

46
00:02:17,130 --> 00:02:21,060
So looking over this first
part, everything here looks OK.

47
00:02:21,060 --> 00:02:22,290
Nothing jumps out.

48
00:02:22,290 --> 00:02:24,450
Let's scroll down a bit.

49
00:02:24,450 --> 00:02:26,220
Again, if we look
over this section,

50
00:02:26,220 --> 00:02:28,710
everything here
looks OK as well.

51
00:02:28,710 --> 00:02:31,290
Here on this section,
there's a few things

52
00:02:31,290 --> 00:02:32,940
we should probably point out.

53
00:02:32,940 --> 00:02:36,870
The first is here, where it says
that the external BGP neighbor

54
00:02:36,870 --> 00:02:39,850
may be up to one hop away.

55
00:02:39,850 --> 00:02:42,810
So that means that the
TTL should theoretically

56
00:02:42,810 --> 00:02:44,700
be sent with a 1.

57
00:02:44,700 --> 00:02:48,300
Now, the next thing says
external BGP neighbor

58
00:02:48,300 --> 00:02:51,660
not configured for
connected checks.

59
00:02:51,660 --> 00:02:55,800
And that's good because these
are peering between loopbacks.

60
00:02:55,800 --> 00:02:58,690
So that would be very
important as well.

61
00:02:58,690 --> 00:03:01,620
Those two things look good.

62
00:03:01,620 --> 00:03:05,910
But if we look at
this line, it says

63
00:03:05,910 --> 00:03:08,670
the connection is ECN-disabled.

64
00:03:08,670 --> 00:03:12,520
And we don't really care
about ECN at this point.

65
00:03:12,520 --> 00:03:17,970
However, it's listing the
minimum incoming TTL as 254,

66
00:03:17,970 --> 00:03:21,280
with the outgoing being 255.

67
00:03:21,280 --> 00:03:24,210
So it's not sending
with a TTL of 1.

68
00:03:24,210 --> 00:03:30,680
It's sending with a 255
and requiring a 254.

69
00:03:30,680 --> 00:03:33,080
This should be a
very clear indicator

70
00:03:33,080 --> 00:03:36,060
that we are doing TTL security.

71
00:03:36,060 --> 00:03:38,270
So that's probably a
good thing to check here.

72
00:03:38,270 --> 00:03:41,780
Let's take a look at
router 7's configuration.

73
00:03:41,780 --> 00:03:44,390
We'll do a show run section BGP.

74
00:03:44,390 --> 00:03:47,780
And we see here we have things
like our disabled connected

75
00:03:47,780 --> 00:03:50,360
check and our update
source loopback 0.

76
00:03:50,360 --> 00:03:55,250
But more importantly, there's
that ttl-security hops 1

77
00:03:55,250 --> 00:03:57,840
that we were just talking about.

78
00:03:57,840 --> 00:04:00,980
So this is, in fact,
using TTL security.

79
00:04:00,980 --> 00:04:04,940
We should probably go verify
that on router 3 then.

80
00:04:04,940 --> 00:04:07,070
So here on router 3,
let's start with our show

81
00:04:07,070 --> 00:04:11,720
IP BGP neighbor
for the 10.7.7.7.

82
00:04:11,720 --> 00:04:14,750
Once again, everything
here looks OK.

83
00:04:14,750 --> 00:04:17,149
Everything here looks OK.

84
00:04:17,149 --> 00:04:21,230
Here we have our two messages
about the external BGP

85
00:04:21,230 --> 00:04:23,280
neighbors not
directly connected.

86
00:04:23,280 --> 00:04:26,030
And that the directly
connected next-hop check

87
00:04:26,030 --> 00:04:28,160
is bypassed for that neighbor.

88
00:04:28,160 --> 00:04:30,200
Those two things
are good, but notice

89
00:04:30,200 --> 00:04:34,530
there is no mention
here of TTL security.

90
00:04:34,530 --> 00:04:37,910
So let's do a show
run router BGP.

91
00:04:37,910 --> 00:04:42,570
And we see, for neighbor
10.7.7.7, that, of course,

92
00:04:42,570 --> 00:04:44,460
we have our normal
configuration--

93
00:04:44,460 --> 00:04:46,910
things like our update source
and our ignore connected

94
00:04:46,910 --> 00:04:47,600
check--

95
00:04:47,600 --> 00:04:51,890
but there's no
TTL security here.

96
00:04:51,890 --> 00:04:55,600
So once again, it looks like
we may have found our problem.

97
00:04:55,600 --> 00:04:59,590
We have one side configured
for TTL security, which

98
00:04:59,590 --> 00:05:05,600
is going to require a
minimum incoming TTL of 254.

99
00:05:05,600 --> 00:05:07,450
And then we have the
other side that's

100
00:05:07,450 --> 00:05:10,990
not configured for TTL
security, and, therefore, would

101
00:05:10,990 --> 00:05:14,620
be originating its
packets with a TTL of 1.

102
00:05:14,620 --> 00:05:17,260
And that's not going to
meet the minimum requirement

103
00:05:17,260 --> 00:05:19,100
on the other side.

104
00:05:19,100 --> 00:05:22,250
So once again, how do we fix it?

105
00:05:22,250 --> 00:05:23,710
Well, we have a
couple of choices,

106
00:05:23,710 --> 00:05:25,960
just like we normally do.

107
00:05:25,960 --> 00:05:27,940
Choice number one,
of course, would

108
00:05:27,940 --> 00:05:30,340
be to remove TTL security.

109
00:05:30,340 --> 00:05:34,130
Assuming that's the problem,
that would definitely fix it.

110
00:05:34,130 --> 00:05:37,210
But once again, going
with the concept that it's

111
00:05:37,210 --> 00:05:38,770
probably here for a reason--

112
00:05:38,770 --> 00:05:41,740
somebody wanted to tell
security on this link--

113
00:05:41,740 --> 00:05:43,900
what are some other choices?

114
00:05:43,900 --> 00:05:47,390
Well, of course, we could
configure router 3 here,

115
00:05:47,390 --> 00:05:52,000
which is missing the TTL
security to have TTL security.

116
00:05:52,000 --> 00:05:54,520
The other choice, though,
is a little interesting,

117
00:05:54,520 --> 00:05:56,440
and we're going to do
it temporarily just

118
00:05:56,440 --> 00:05:57,880
to show you that it works.

119
00:05:57,880 --> 00:05:59,950
I would not recommend
this, but it's just

120
00:05:59,950 --> 00:06:05,360
an interesting exercise in how
TTL security actually works.

121
00:06:05,360 --> 00:06:08,230
Remember that what
router 7 is expecting

122
00:06:08,230 --> 00:06:12,070
is a TTL of 254 when
the packet gets there.

123
00:06:12,070 --> 00:06:14,990
That's really all
we have to provide.

124
00:06:14,990 --> 00:06:21,550
So if I were to tell router
3 to use EBGP Multihop

125
00:06:21,550 --> 00:06:26,710
and set the TTL to 255, that
would meet the requirement

126
00:06:26,710 --> 00:06:30,500
on router 7, and this
neighborship would come up.

127
00:06:30,500 --> 00:06:35,230
You're, in a way, tricking or
circumventing the TTL security

128
00:06:35,230 --> 00:06:36,200
at that point.

129
00:06:36,200 --> 00:06:38,930
And again, like I said,
I wouldn't recommend it.

130
00:06:38,930 --> 00:06:41,410
But I think it's a good
exercise to go through

131
00:06:41,410 --> 00:06:45,400
just so we can see that that's
really how TTL security works.

132
00:06:45,400 --> 00:06:48,010
And yes, if we were
directly connected,

133
00:06:48,010 --> 00:06:52,150
we could do this and
bypass it or circumvent it.

134
00:06:52,150 --> 00:06:54,790
Keep in mind, of course, if
we weren't directly connected,

135
00:06:54,790 --> 00:06:56,200
this wouldn't work, right?

136
00:06:56,200 --> 00:06:58,270
If we were really
three hops away,

137
00:06:58,270 --> 00:07:01,810
it doesn't matter what we
changed EBGP Multihop to.

138
00:07:01,810 --> 00:07:06,580
So although we're working
around the TTL security thing,

139
00:07:06,580 --> 00:07:10,630
I don't want you to think that
this is a vulnerability in TTL

140
00:07:10,630 --> 00:07:11,440
security.

141
00:07:11,440 --> 00:07:14,770
TTL security is still
doing its job as it should.

142
00:07:14,770 --> 00:07:17,380
We're just providing
its requirement

143
00:07:17,380 --> 00:07:20,958
in a unique or unusual way.

144
00:07:20,958 --> 00:07:23,500
So like I said, let's go ahead
and configure that real quick.

145
00:07:23,500 --> 00:07:25,042
Just make sure the
neighbors come up.

146
00:07:25,042 --> 00:07:27,110
And then, of course,
we'll fix it right.

147
00:07:27,110 --> 00:07:29,900
But I do want you to
see that this does work.

148
00:07:29,900 --> 00:07:33,550
So on router 3 here, let's
go into configuration mode.

149
00:07:33,550 --> 00:07:40,090
Router BGP 100, neighbor
10.7.7.7, EBGP Multihop.

150
00:07:40,090 --> 00:07:43,960
And again, we'll set
it to the maximum 255.

151
00:07:43,960 --> 00:07:48,070
That way, it'll meet the
requirement that's on router 7.

152
00:07:48,070 --> 00:07:50,080
We'll commit that change.

153
00:07:50,080 --> 00:07:52,780
We won't actually exit
configuration this time

154
00:07:52,780 --> 00:07:55,820
because we're going to
change it in just a moment.

155
00:07:55,820 --> 00:07:58,910
Now it will take these neighbors
just a few minutes to come up.

156
00:07:58,910 --> 00:08:02,270
So let's fast forward
until that's done.

157
00:08:02,270 --> 00:08:03,920
So it has been a few minutes.

158
00:08:03,920 --> 00:08:08,600
And we can see that now
10.7.7.7 is up and running.

159
00:08:08,600 --> 00:08:11,180
It came up about 33 seconds ago.

160
00:08:11,180 --> 00:08:17,570
So EBGP Multihop does, in
fact, work with TTL security.

161
00:08:17,570 --> 00:08:20,680
Now again, not exactly how
we probably want to leave it,

162
00:08:20,680 --> 00:08:22,700
but it does work.

163
00:08:22,700 --> 00:08:26,050
So let's go ahead and
take this feature off.

164
00:08:26,050 --> 00:08:27,852
We'll commit that change.

165
00:08:27,852 --> 00:08:29,560
And then we'll wait
for that neighborship

166
00:08:29,560 --> 00:08:32,919
to go back down just to prove
that our other solution works

167
00:08:32,919 --> 00:08:34,130
as well.

168
00:08:34,130 --> 00:08:36,370
So we're now past
the whole time.

169
00:08:36,370 --> 00:08:41,289
And we see that neighbor
10.7.7.7 is back to the state

170
00:08:41,289 --> 00:08:44,720
OpenConfirm, which means
we're back to not working.

171
00:08:44,720 --> 00:08:48,100
So now let's put in what we
would call probably the best

172
00:08:48,100 --> 00:08:49,610
solution.

173
00:08:49,610 --> 00:08:52,330
So we'll turn on TTL security.

174
00:08:52,330 --> 00:08:54,380
We'll commit the changes.

175
00:08:54,380 --> 00:08:57,010
And once again, we will have
to wait just a few minutes

176
00:08:57,010 --> 00:08:59,185
for this to come back up.

177
00:08:59,185 --> 00:09:00,560
So it's been a
couple of minutes.

178
00:09:00,560 --> 00:09:03,380
Let's go ahead and check
our neighborship again.

179
00:09:03,380 --> 00:09:07,340
And we see that
10.7.7.7 is back up.

180
00:09:07,340 --> 00:09:09,790
It's been up for
about 30 seconds.

181
00:09:09,790 --> 00:09:13,780
We'll, of course, verify
this on router 7 as well.

182
00:09:13,780 --> 00:09:16,120
We'll do our show
IP BGP summary.

183
00:09:16,120 --> 00:09:20,910
And we see that 10.3.3.3
is now up as a neighbor.

184
00:09:20,910 --> 00:09:24,810
So that verifies that TTL
security was the problem,

185
00:09:24,810 --> 00:09:28,250
and we have solved this ticket.

186
00:09:28,250 --> 00:09:33,490
So in this video, we saw an
issue related to TTL security.

187
00:09:33,490 --> 00:09:37,420
Remember that the way
TTL security works is

188
00:09:37,420 --> 00:09:41,590
it sends all of its
updates with a TTL of 255.

189
00:09:41,590 --> 00:09:44,590
And then, it has a
minimum TTL that's

190
00:09:44,590 --> 00:09:47,260
required on the
packets it receives.

191
00:09:47,260 --> 00:09:51,160
As long as the neighbor
is sending its BGP updates

192
00:09:51,160 --> 00:09:55,730
with the appropriate TTL,
then everything will be fine.

193
00:09:55,730 --> 00:09:59,560
Now the way you get the neighbor
to do that is, well, as we saw,

194
00:09:59,560 --> 00:10:00,800
one of two ways.

195
00:10:00,800 --> 00:10:04,690
We could theoretically
use EBGP Multihop.

196
00:10:04,690 --> 00:10:07,180
Again, probably not
the best solution,

197
00:10:07,180 --> 00:10:09,460
but we saw that it
does, in fact, work.

198
00:10:09,460 --> 00:10:15,140
The best thing, of course, is to
run TTL security on both sides.

199
00:10:15,140 --> 00:10:17,170
I hope this has been
informative for you,

200
00:10:17,170 --> 00:10:20,250
and I'd like to thank
you for viewing.

201
00:10:20,250 --> 00:10:24,000