1
00:00:00,000 --> 00:00:03,864
[AUDIO LOGO]

2
00:00:03,864 --> 00:00:13,060

3
00:00:13,060 --> 00:00:15,220
Next up is ticket 5.

4
00:00:15,220 --> 00:00:19,740
This, once again, is
on OSPF topology 3.

5
00:00:19,740 --> 00:00:23,350
And, well, the ticket itself is
going to sound fairly familiar

6
00:00:23,350 --> 00:00:24,200
as well.

7
00:00:24,200 --> 00:00:26,410
This time, we're
talking about R9,

8
00:00:26,410 --> 00:00:30,130
but we're still with
that-- has no OSPF routes

9
00:00:30,130 --> 00:00:31,520
in its routing table.

10
00:00:31,520 --> 00:00:34,570
So of course, we will
start on router 9.

11
00:00:34,570 --> 00:00:37,000
I'm sure we'll do some
exploring on router 2

12
00:00:37,000 --> 00:00:38,960
as well, if not
even beyond that.

13
00:00:38,960 --> 00:00:42,730
But again, we'll start off
with our main focus here being

14
00:00:42,730 --> 00:00:46,420
on this area between R2 and R9.

15
00:00:46,420 --> 00:00:48,790
So let's jump into the CLI.

16
00:00:48,790 --> 00:00:51,730
We'll start on router 9,
do some investigation,

17
00:00:51,730 --> 00:00:54,630
and go from there.

18
00:00:54,630 --> 00:00:58,610
So starting off on router 9
with a show ip route ospf,

19
00:00:58,610 --> 00:01:02,450
we see that, as the ticket said,
we have no OSPF routes here

20
00:01:02,450 --> 00:01:03,620
on router 9.

21
00:01:03,620 --> 00:01:06,350
So let's look at some
of our normal things.

22
00:01:06,350 --> 00:01:09,590
Let's do a show ip
ospf interface brief.

23
00:01:09,590 --> 00:01:11,460
According to this,
we are, in fact,

24
00:01:11,460 --> 00:01:14,120
running OSPF on the
Gigabit 2 interface.

25
00:01:14,120 --> 00:01:16,760
It's in area 1000, like
it's supposed to be.

26
00:01:16,760 --> 00:01:19,670
It does have the correct
IP address and subnet

27
00:01:19,670 --> 00:01:24,453
mask, configured for point to
point, but has zero neighbors.

28
00:01:24,453 --> 00:01:26,120
Now, we could, of
course, follow this up

29
00:01:26,120 --> 00:01:28,100
with a show ip ospf neighbors.

30
00:01:28,100 --> 00:01:29,810
We already know this
is going to be blank

31
00:01:29,810 --> 00:01:32,870
because all of our interfaces
have zero neighbors.

32
00:01:32,870 --> 00:01:37,040
So again, things look
relatively OK here on router 9.

33
00:01:37,040 --> 00:01:41,290
So let's do our preliminary
checks over on router 2.

34
00:01:41,290 --> 00:01:43,990
So a show ip ospf
interface brief.

35
00:01:43,990 --> 00:01:48,490
We see that we again have OSPF
enabled on Gigabit 0/0/0/2,

36
00:01:48,490 --> 00:01:49,870
which is correct.

37
00:01:49,870 --> 00:01:51,700
It's in area 1000.

38
00:01:51,700 --> 00:01:53,830
It has the correct
IP address and mask.

39
00:01:53,830 --> 00:01:56,200
It's also configured
point to point.

40
00:01:56,200 --> 00:02:00,410
And it has no neighbors, at
least no complete neighbors.

41
00:02:00,410 --> 00:02:05,350
But notice that we do have
one set to connecting.

42
00:02:05,350 --> 00:02:10,030
Let's take a look at our
show ip ospf neighbors.

43
00:02:10,030 --> 00:02:15,580
And what we see here is it
actually knows about router 9.

44
00:02:15,580 --> 00:02:18,380
It's currently in
the init phase,

45
00:02:18,380 --> 00:02:20,240
which is why it's connecting.

46
00:02:20,240 --> 00:02:23,890
It's not full, but it is in
the process of connecting.

47
00:02:23,890 --> 00:02:27,380
Now, if we give that just
a few seconds, try again,

48
00:02:27,380 --> 00:02:32,020
see if it's moving past init in
any way, we see that it's not.

49
00:02:32,020 --> 00:02:37,350
It is currently just stuck here
in the initialization phase.

50
00:02:37,350 --> 00:02:41,630
So this actually brings up an
interesting situation as far

51
00:02:41,630 --> 00:02:42,980
as troubleshooting.

52
00:02:42,980 --> 00:02:48,080
The fact that router 2
sees router 9 but router 9

53
00:02:48,080 --> 00:02:53,510
sees nothing, indicates
that the hello from router 9

54
00:02:53,510 --> 00:02:57,440
is getting to router
2 because we see it,

55
00:02:57,440 --> 00:02:59,870
and we recognize
that, hey, as long

56
00:02:59,870 --> 00:03:03,240
as we complete this process,
this is going to be a neighbor.

57
00:03:03,240 --> 00:03:07,880
However, it looks like
the hello from router 2

58
00:03:07,880 --> 00:03:10,730
is not getting to router 9.

59
00:03:10,730 --> 00:03:14,610
So it looks like we have
hellos going in one direction.

60
00:03:14,610 --> 00:03:17,070
Let's investigate this
just a little bit further.

61
00:03:17,070 --> 00:03:21,230
Let's take a look at some debugs
and maybe even some Wireshark.

62
00:03:21,230 --> 00:03:24,890
So let's start with
a debug ospf 1.

63
00:03:24,890 --> 00:03:26,810
And let's start by
debugging the hellos.

64
00:03:26,810 --> 00:03:31,950

65
00:03:31,950 --> 00:03:35,670
So what we see right here is
that we're receiving a hello

66
00:03:35,670 --> 00:03:41,740
from 10.9.9.9 in area 1000 on
our Gigabit 0/0/0/2 interface.

67
00:03:41,740 --> 00:03:46,020
So as we suspected, it's
receiving this hello.

68
00:03:46,020 --> 00:03:48,180
And then we see a little
bit further down here

69
00:03:48,180 --> 00:03:53,760
that it's sending the hello
to 224.0.0.5 in area 1000

70
00:03:53,760 --> 00:03:55,780
on that same interface.

71
00:03:55,780 --> 00:04:00,210
So we see that router 2 is
sending and receiving hellos.

72
00:04:00,210 --> 00:04:02,490
Let's take a look at router 9.

73
00:04:02,490 --> 00:04:04,800
The command is a little bit
different here on router 9.

74
00:04:04,800 --> 00:04:07,440
It's debug ip ospf hello.

75
00:04:07,440 --> 00:04:09,660
We'll turn on terminal monitor.

76
00:04:09,660 --> 00:04:12,160
We'll let this run here
for just a few moments.

77
00:04:12,160 --> 00:04:16,470
So what we see here is clearly
we are sending hellos, which

78
00:04:16,470 --> 00:04:18,300
we already knew,
because of course,

79
00:04:18,300 --> 00:04:20,040
router 2 is receiving them.

80
00:04:20,040 --> 00:04:24,180
But we're not receiving
any hellos from router 2.

81
00:04:24,180 --> 00:04:29,280
So let's actually take a look
at what's going on in Wireshark.

82
00:04:29,280 --> 00:04:31,710
So here we have a
Wireshark capture

83
00:04:31,710 --> 00:04:35,700
going on, on router 9's
Gigabit 2 interface.

84
00:04:35,700 --> 00:04:38,290
And let's go ahead and
stop this just for a second

85
00:04:38,290 --> 00:04:39,480
so we can take a look at it.

86
00:04:39,480 --> 00:04:41,940
Now, we can see that we
have some OSPF version

87
00:04:41,940 --> 00:04:43,320
3 packets here.

88
00:04:43,320 --> 00:04:47,280
These would be the
ones going to ff02::5.

89
00:04:47,280 --> 00:04:51,960
We're of course concerned about
the ones going to 224.0.0.5,

90
00:04:51,960 --> 00:04:54,540
because that's OSPF version 2.

91
00:04:54,540 --> 00:04:58,420
So if we were to take a look
at packet 6 here, for example,

92
00:04:58,420 --> 00:05:05,100
we can see that this frame is
in fact coming from 50.1.29.2.

93
00:05:05,100 --> 00:05:06,195
So that's router 2.

94
00:05:06,195 --> 00:05:08,840

95
00:05:08,840 --> 00:05:11,670
So what is this
actually telling us?

96
00:05:11,670 --> 00:05:15,860
This is telling us that the
packet is actually arriving

97
00:05:15,860 --> 00:05:18,680
on the interface on router 9.

98
00:05:18,680 --> 00:05:23,030
So something's happening
between the packet arriving

99
00:05:23,030 --> 00:05:27,050
on the actual interface and
the packet being received

100
00:05:27,050 --> 00:05:30,200
by the OSPF process,
because it's not

101
00:05:30,200 --> 00:05:32,840
showing up in our OSPF debugs.

102
00:05:32,840 --> 00:05:35,450
Now, the first thing
that comes to mind there

103
00:05:35,450 --> 00:05:38,600
is something that would
actually stop the packets

104
00:05:38,600 --> 00:05:43,730
at the ingress interface, which
would generally be an ACL.

105
00:05:43,730 --> 00:05:45,830
Let's go ahead and
test that theory

106
00:05:45,830 --> 00:05:50,660
and see if there's an
access list on router 9.

107
00:05:50,660 --> 00:05:52,920
By the way, before
we actually do that,

108
00:05:52,920 --> 00:05:55,280
let's do one other
test that can be

109
00:05:55,280 --> 00:05:57,630
an interesting
troubleshooting tool as well.

110
00:05:57,630 --> 00:06:00,200
We probably don't need
it in this exact case,

111
00:06:00,200 --> 00:06:02,600
but it's a good tool to
have in your toolbox.

112
00:06:02,600 --> 00:06:07,620
You can actually ping the OSPF
version 2 multicast address.

113
00:06:07,620 --> 00:06:11,600
And what should happen here
is we should get a response

114
00:06:11,600 --> 00:06:13,560
from all of our neighbors.

115
00:06:13,560 --> 00:06:16,430
But notice, the only device
we got a response from

116
00:06:16,430 --> 00:06:17,660
was ourself.

117
00:06:17,660 --> 00:06:20,810
So once again, this is
a pretty good indicator

118
00:06:20,810 --> 00:06:23,670
that we're not receiving
the traffic properly.

119
00:06:23,670 --> 00:06:26,790
So let's go ahead and take
a look at that interface.

120
00:06:26,790 --> 00:06:28,940
Let's do a show ip interface.

121
00:06:28,940 --> 00:06:31,700
And again, this is
all about Gigabit 2.

122
00:06:31,700 --> 00:06:36,620
As suspected, we have
an inbound access list

123
00:06:36,620 --> 00:06:43,080
called ACL_FILTER applied
to this Gigabit 2 interface.

124
00:06:43,080 --> 00:06:44,990
So let's take a look
at that access list.

125
00:06:44,990 --> 00:06:47,990

126
00:06:47,990 --> 00:06:49,980
We'll just do a
show ip access-list.

127
00:06:49,980 --> 00:06:52,110
This is actually the
only one on this device.

128
00:06:52,110 --> 00:06:57,360
We see that it's permitting
UDP if it equals domain--

129
00:06:57,360 --> 00:06:59,480
so DNS traffic.

130
00:06:59,480 --> 00:07:04,340
It's permitting TCP for
both normal WWW traffic

131
00:07:04,340 --> 00:07:08,390
as well as SSL
traffic on port 443.

132
00:07:08,390 --> 00:07:12,430
But it's not allowing OSPF.

133
00:07:12,430 --> 00:07:16,060
Keep in mind that
with an inbound ACL,

134
00:07:16,060 --> 00:07:19,630
it blocks all traffic
that's not permitted.

135
00:07:19,630 --> 00:07:24,140
Remember that ACLs have a
default deny any at the end.

136
00:07:24,140 --> 00:07:26,950
So even though it's not
shown here, it's there.

137
00:07:26,950 --> 00:07:31,000
And it denies all traffic
that's not permitted.

138
00:07:31,000 --> 00:07:34,900
With an outbound ACL, this
would actually be different

139
00:07:34,900 --> 00:07:38,740
because the router will not
filter the traffic generated

140
00:07:38,740 --> 00:07:40,180
by the router itself.

141
00:07:40,180 --> 00:07:42,910
But with an inbound
ACL, well, it

142
00:07:42,910 --> 00:07:44,630
doesn't know who
generated the traffic.

143
00:07:44,630 --> 00:07:46,250
It's just incoming traffic.

144
00:07:46,250 --> 00:07:48,140
So it's going to
block all of it.

145
00:07:48,140 --> 00:07:52,690
So once again, we have a couple
of choices on how to fix this.

146
00:07:52,690 --> 00:07:56,210
One, of course, would be
simply to take the ACL off.

147
00:07:56,210 --> 00:07:58,900
But if we're looking at
this, we could probably

148
00:07:58,900 --> 00:08:03,100
make a reasonable design
decision that, well, whoever

149
00:08:03,100 --> 00:08:05,620
put this here probably
actually wanted it here

150
00:08:05,620 --> 00:08:07,430
and wanted it working.

151
00:08:07,430 --> 00:08:11,360
So maybe we should just modify
it to allow the things we want.

152
00:08:11,360 --> 00:08:13,990
So let's just go ahead and
modify this access list.

153
00:08:13,990 --> 00:08:15,765
We'll allow OSPF.

154
00:08:15,765 --> 00:08:17,140
And while we're
at it, we'll just

155
00:08:17,140 --> 00:08:20,590
allow pings so that our little
troubleshooting tool there

156
00:08:20,590 --> 00:08:22,430
can work properly as well.

157
00:08:22,430 --> 00:08:24,790
So let's just jump in
here, change the ACL,

158
00:08:24,790 --> 00:08:27,920
and see if that
solves our problem.

159
00:08:27,920 --> 00:08:30,260
We'll just go into
configuration mode--

160
00:08:30,260 --> 00:08:36,480
ip access-list extended, and
this is called ACL_FILTER.

161
00:08:36,480 --> 00:08:39,890
Now, we could theoretically
put these rules anywhere.

162
00:08:39,890 --> 00:08:42,470
But I think we should have
a goal of putting them

163
00:08:42,470 --> 00:08:46,350
towards the top since they're
actual network functionality.

164
00:08:46,350 --> 00:08:50,850
So maybe we say line 5 permit.

165
00:08:50,850 --> 00:08:53,070
Let's go and do the pings first.

166
00:08:53,070 --> 00:08:56,760
So we'll just do icmp
from anyone to anyone.

167
00:08:56,760 --> 00:08:57,938
And then we'll say--

168
00:08:57,938 --> 00:08:58,980
it's not going to matter.

169
00:08:58,980 --> 00:09:00,450
You'll see why in just a moment.

170
00:09:00,450 --> 00:09:07,270
But let's do 8, and we'll say
to permit ospf from any to any.

171
00:09:07,270 --> 00:09:09,610
Now, if this actually
solves the problem,

172
00:09:09,610 --> 00:09:12,550
OSPF should come up
pretty much right away.

173
00:09:12,550 --> 00:09:14,380
We can see here that it has.

174
00:09:14,380 --> 00:09:17,500
However, one thing
that I want to fix here

175
00:09:17,500 --> 00:09:22,510
is if I do a do show
ip access-list now,

176
00:09:22,510 --> 00:09:26,200
we can see that we have
10 matches on OSPF.

177
00:09:26,200 --> 00:09:28,930
But our line numbering
is a little weird.

178
00:09:28,930 --> 00:09:32,570
It's 5, 8, then 10,
20, 30, nice and neat.

179
00:09:32,570 --> 00:09:34,720
So one of the things
you can do here

180
00:09:34,720 --> 00:09:37,640
is you can actually
resequence this.

181
00:09:37,640 --> 00:09:40,990
So if we drop out of the
ACL configuration mode,

182
00:09:40,990 --> 00:09:45,960
we can say, ip
access-list resequence.

183
00:09:45,960 --> 00:09:48,330
Then list what we
want to resequence--

184
00:09:48,330 --> 00:09:50,440
ACL_FILTER.

185
00:09:50,440 --> 00:09:52,270
Then you give it
a starting number.

186
00:09:52,270 --> 00:09:53,910
So let's start at 10.

187
00:09:53,910 --> 00:09:56,230
And then you give it
an increment number.

188
00:09:56,230 --> 00:10:00,510
So let's start at
10 and jump by tens.

189
00:10:00,510 --> 00:10:05,830
And then if we say,
show ip access-list,

190
00:10:05,830 --> 00:10:07,900
notice that they're
now nicely numbered.

191
00:10:07,900 --> 00:10:09,730
10, 20, 30, 40, 50.

192
00:10:09,730 --> 00:10:13,360
This is nice if we've had to
add to the list over time.

193
00:10:13,360 --> 00:10:14,890
It started to get
a little crowded

194
00:10:14,890 --> 00:10:16,040
in our numbering scheme.

195
00:10:16,040 --> 00:10:18,460
And we need to resequence it.

196
00:10:18,460 --> 00:10:23,200
One quick note on renumbering
or resequencing your ACLs,

197
00:10:23,200 --> 00:10:26,020
make sure everybody
in the management team

198
00:10:26,020 --> 00:10:27,670
knows that this is happening.

199
00:10:27,670 --> 00:10:30,100
Put it in your documentation,
in your log entries,

200
00:10:30,100 --> 00:10:35,080
whatever, because particularly,
if you're logging ACL entries--

201
00:10:35,080 --> 00:10:37,870
which were not, in this
case, but if we were logging

202
00:10:37,870 --> 00:10:41,570
these entries, all of a sudden,
the logging is going to change.

203
00:10:41,570 --> 00:10:43,990
In other words, the
traffic that was permitted

204
00:10:43,990 --> 00:10:48,550
by ACL entry 8 a
moment ago is now

205
00:10:48,550 --> 00:10:51,820
being permitted by ACL list 20.

206
00:10:51,820 --> 00:10:55,270
So in your logs, that might
look a little interesting

207
00:10:55,270 --> 00:10:58,880
why all of a sudden things
changed rule numbers and such.

208
00:10:58,880 --> 00:11:02,320
So again, just keep
track of what you change.

209
00:11:02,320 --> 00:11:04,180
Always document
this kind of thing.

210
00:11:04,180 --> 00:11:06,850
And make sure everybody
knows it happened.

211
00:11:06,850 --> 00:11:09,250
Otherwise, it could,
again, make troubleshooting

212
00:11:09,250 --> 00:11:11,470
or trying to look through
historical events--

213
00:11:11,470 --> 00:11:14,710
could get really interesting
if you don't have documentation

214
00:11:14,710 --> 00:11:16,270
that this was resequenced.

215
00:11:16,270 --> 00:11:17,780
Just keep that in mind.

216
00:11:17,780 --> 00:11:20,740
So now that we got the console
message that this came up,

217
00:11:20,740 --> 00:11:22,990
we probably don't
really need to check.

218
00:11:22,990 --> 00:11:25,340
But you know that
we always should.

219
00:11:25,340 --> 00:11:27,550
Let's go ahead and exit
out of configuration mode.

220
00:11:27,550 --> 00:11:29,960
We'll do a show
ip ospf neighbor.

221
00:11:29,960 --> 00:11:32,050
We see that our
neighborship with router 2

222
00:11:32,050 --> 00:11:35,221
is now up and full.

223
00:11:35,221 --> 00:11:38,650
If we do a show ip
route ospf, we now

224
00:11:38,650 --> 00:11:41,830
see that we are getting
entries in our routing table

225
00:11:41,830 --> 00:11:43,960
where we weren't
getting them before.

226
00:11:43,960 --> 00:11:46,760
And just to revisit our
troubleshooting tool,

227
00:11:46,760 --> 00:11:50,110
this is what we should have
gotten in a proper environment

228
00:11:50,110 --> 00:11:53,350
when we pinged the 224.0.0.5.

229
00:11:53,350 --> 00:11:56,080
Notice that we're now
getting a response

230
00:11:56,080 --> 00:12:00,910
both from ourselves and
from our neighbor, router 2.

231
00:12:00,910 --> 00:12:04,330
So this is what this should
look like if it is working.

232
00:12:04,330 --> 00:12:07,660
And of course, we saw earlier
what it looks like if it's not

233
00:12:07,660 --> 00:12:10,070
working properly.

234
00:12:10,070 --> 00:12:13,180
So in this video,
we saw an example

235
00:12:13,180 --> 00:12:17,800
of an access list,
particularly an inbound ACL,

236
00:12:17,800 --> 00:12:21,310
blocking the OSPF traffic,
preventing the neighbor

237
00:12:21,310 --> 00:12:22,610
from coming up.

238
00:12:22,610 --> 00:12:25,180
Now, as you saw, this
one's a little trickier

239
00:12:25,180 --> 00:12:27,700
to troubleshoot than some
of the other options we've

240
00:12:27,700 --> 00:12:29,230
looked at so far.

241
00:12:29,230 --> 00:12:31,540
We saw that all the
hellos looked OK.

242
00:12:31,540 --> 00:12:34,720
All the parameters were
configured OK on the devices.

243
00:12:34,720 --> 00:12:37,510
Just the neighbors
weren't coming up.

244
00:12:37,510 --> 00:12:39,140
Debugs helped quite a bit.

245
00:12:39,140 --> 00:12:42,160
We could see that one side was
both sending and receiving,

246
00:12:42,160 --> 00:12:44,830
while the other side
was only sending.

247
00:12:44,830 --> 00:12:47,500
Then, of course, Wireshark
helped us out a bit

248
00:12:47,500 --> 00:12:52,130
there by seeing it's actually
receiving the traffic.

249
00:12:52,130 --> 00:12:55,710
It's just not getting
to the OSPF process.

250
00:12:55,710 --> 00:12:58,140
That pointed us in a
pretty good direction

251
00:12:58,140 --> 00:13:01,720
to suspect that it was
an ACL to begin with.

252
00:13:01,720 --> 00:13:05,550
Of course, once we fixed
the ACL and permitted OSPF,

253
00:13:05,550 --> 00:13:08,670
then our neighbors came
right up as they should.

254
00:13:08,670 --> 00:13:10,770
I hope this has been
informative for you,

255
00:13:10,770 --> 00:13:13,790
and I'd like to thank
you for viewing.

256
00:13:13,790 --> 00:13:17,000