WEBVTT

1
00:00.000 --> 00:08.640
So in our previous video, we got tracked objects working with IPSLA, but we

2
00:08.640 --> 00:09.600
also witnessed

3
00:09.600 --> 00:14.970
the problem when we have random issues on the WAN that this can actually cause

4
00:14.970 --> 00:15.160
things

5
00:15.160 --> 00:20.990
like VRRP or really the tracked object itself to flap between up and down and

6
00:20.990 --> 00:22.280
causing things

7
00:22.280 --> 00:27.840
like VRRP to switch their state back and forth between the devices continuously

8
00:27.840 --> 00:27.880
.

9
00:27.880 --> 00:32.840
So what might be a better plan is to have it failover just once.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

9
00:27.880 --> 00:32.840
So what might be a better plan is to have it failover just once.

10
00:32.840 --> 00:38.080
For that, we can implement something called EEM or embedded event manager.

11
00:38.080 --> 00:46.090
We'll have EEM watch the IPSLA, then we'll have that trip a tracked object, but

12
00:46.090 --> 00:46.200
that

13
00:46.200 --> 00:49.960
tracked object will then stay down.

14
00:49.960 --> 00:51.400
That's the whole point of this.

15
00:51.400 --> 00:55.320
We're only going to switch it from up to down once.

16
00:55.320 --> 00:59.560
And of course, that will still cause the failover of VRRP.

17
00:59.560 --> 01:05.360
So let's go ahead and jump in, take a look at how this works.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

17
00:59.560 --> 01:05.360
So let's go ahead and jump in, take a look at how this works.

18
01:05.360 --> 01:08.650
So the first thing we want to look at is something else that might help a

19
01:08.650 --> 01:09.440
little bit with this

20
01:09.440 --> 01:12.080
flapping situation that we saw.

21
01:12.080 --> 01:16.590
If we actually go in to say tracked object one, one of the options we saw

22
01:16.590 --> 01:17.440
earlier but

23
01:17.440 --> 01:21.240
didn't actually look at is delay.

24
01:21.240 --> 01:28.090
And on the delay, you have the option of changing both the down and the uptimer

25
01:28.090 --> 01:28.120
.

26
01:28.120 --> 01:34.030
So we could, for example, say to set the down timer and this is in seconds, we
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

26
01:28.120 --> 01:34.030
So we could, for example, say to set the down timer and this is in seconds, we

27
01:34.030 --> 01:34.880
could say

28
01:34.880 --> 01:40.730
don't take the tracked object down until the problem has existed for 10 seconds

29
01:40.730 --> 01:41.040
and

30
01:41.040 --> 01:47.060
don't bring it back up until it has been fixed for, and again, we could go up

31
01:47.060 --> 01:48.160
to 180

32
01:48.160 --> 01:49.160
seconds here.

33
01:49.160 --> 01:54.910
Maybe we don't have it come up for one minute after the problem has been solved

34
01:54.910 --> 01:54.960
.

35
01:54.960 --> 01:59.510
Now we're not going to go ahead and configure this because at the end of the

36
01:59.510 --> 02:00.360
day, all of
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

36
01:59.510 --> 02:00.360
day, all of

37
02:00.360 --> 02:06.000
this is really going to do in most cases is slow the problem down.

38
02:06.000 --> 02:10.270
What this is really for in all honesty is to allow things like routing

39
02:10.270 --> 02:11.240
protocols and

40
02:11.240 --> 02:16.080
such to stabilize before we bring a tracked object back up.

41
02:16.080 --> 02:17.760
That's what this is really meant for.

42
02:17.760 --> 02:21.920
So although it might help in some situations with our problem, this isn't

43
02:21.920 --> 02:22.600
really what we're

44
02:22.600 --> 02:23.680
looking for.

45
02:23.680 --> 02:27.880
So let's go ahead and take a look at how to solve this with EEM.

46
02:27.880 --> 02:30.950
So instead what we're going to do is we're actually going to get rid of our
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

46
02:27.880 --> 02:30.950
So instead what we're going to do is we're actually going to get rid of our

47
02:30.950 --> 02:31.840
tracked objects

48
02:31.840 --> 02:32.840
for now.

49
02:32.840 --> 02:35.600
We're going to recreate them, but we're going to do them differently.

50
02:35.600 --> 02:38.840
So we'll just get rid of track one and track two.

51
02:38.840 --> 02:44.330
And the other thing we want to do is go back into our gigabit one interface and

52
02:44.330 --> 02:44.600
tell it

53
02:44.600 --> 02:47.960
not to track tracked objects to anymore.

54
02:47.960 --> 02:54.340
So again, ultimately what we're going to be doing here is using IPSLA 2 to

55
02:54.340 --> 02:56.760
trigger EEM.

56
02:56.760 --> 02:59.200
But here's a small problem.

57
02:59.200 --> 03:04.560
If we do a show run section SLA, we can see here that we still have our SLA 1
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

57
02:59.200 --> 03:04.560
If we do a show run section SLA, we can see here that we still have our SLA 1

58
03:04.560 --> 03:05.640
and our SLA

59
03:05.640 --> 03:06.640
2.

60
03:06.640 --> 03:12.710
However, this isn't a complete configuration if we want to send this

61
03:12.710 --> 03:14.440
information to EEM.

62
03:14.440 --> 03:20.530
In order to do that, we have to set up what's called an IPSLA reaction

63
03:20.530 --> 03:22.440
configuration.

64
03:22.440 --> 03:27.480
We're going to set up a reaction configuration for entry number two.

65
03:27.480 --> 03:32.030
For number two, we're going to say to react and we can choose what we want it
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

65
03:27.480 --> 03:32.030
For number two, we're going to say to react and we can choose what we want it

66
03:32.030 --> 03:32.640
to react

67
03:32.640 --> 03:34.000
to.

68
03:34.000 --> 03:38.660
What we're going to do, this is of course a jitter probe and we're going to

69
03:38.660 --> 03:39.320
look at the

70
03:39.320 --> 03:42.880
source to destination average jitter.

71
03:42.880 --> 03:47.720
The next thing we have to do is tell it what threshold to use.

72
03:47.720 --> 03:53.240
So we're going to say that the threshold value at what number should we trigger

73
03:53.240 --> 03:53.560
this?

74
03:53.560 --> 03:58.320
Now we know from our previous videos that when we implemented our jitter

75
03:58.320 --> 03:58.960
because of the

76
03:58.960 --> 04:03.440
randomness, it was only hitting around 68 or so.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

76
03:58.960 --> 04:03.440
randomness, it was only hitting around 68 or so.

77
04:03.440 --> 04:07.320
So let's just set it to 60 so we can make sure that we trigger this.

78
04:07.320 --> 04:13.640
Then we'll say to stop triggering or the low limit, let's set this to around 30

79
04:13.640 --> 04:14.280
so it stops

80
04:14.280 --> 04:19.120
triggering it when we get back down to what we'd say are our normal numbers.

81
04:19.120 --> 04:21.920
We also want to set the threshold type.

82
04:21.920 --> 04:25.160
You can see the choice here is we can do average consecutive.

83
04:25.160 --> 04:29.770
We could do immediate too so it triggers instantly or we can say however many

84
04:29.770 --> 04:30.560
out of however
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

84
04:29.770 --> 04:30.560
out of however

85
04:30.560 --> 04:31.560
many occurrences.

86
04:31.560 --> 04:37.180
We'll just go ahead and set this to an average and we'll set it to an average

87
04:37.180 --> 04:38.280
of say five.

88
04:38.280 --> 04:44.460
So if the average over five instances is above 60, then it's going to create

89
04:44.460 --> 04:46.760
this react event.

90
04:46.760 --> 04:50.200
The other thing we have to do is actually enable this.

91
04:50.200 --> 04:56.640
So you have to say IPSLA enable react alerts.

92
04:56.640 --> 04:59.080
So it actually creates them.

93
04:59.080 --> 05:03.920
Now EM can be configured to support this.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

93
04:59.080 --> 05:03.920
Now EM can be configured to support this.

94
05:03.920 --> 05:09.040
However, before we put our EM applet in, let's go ahead and make our tracked

95
05:09.040 --> 05:09.800
object.

96
05:09.800 --> 05:14.060
So we'll just use track one again, but this time we're going to do it a little

97
05:14.060 --> 05:14.840
different.

98
05:14.840 --> 05:18.840
We're going to create something called a stub object.

99
05:18.840 --> 05:24.600
A stub object is something that we as administrators can set up or set down

100
05:24.600 --> 05:26.160
however we want.

101
05:26.160 --> 05:31.520
We can do it manually or we can trigger it say with EM.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

101
05:26.160 --> 05:31.520
We can do it manually or we can trigger it say with EM.

102
05:31.520 --> 05:35.930
So we're going to go into the stub object and then we're going to say that the

103
05:35.930 --> 05:36.560
default

104
05:36.560 --> 05:40.280
state for this is going to be up.

105
05:40.280 --> 05:44.420
So we'll say the default state is up if we say do show track.

106
05:44.420 --> 05:49.280
We should now have tracked object one, which is a stub object, meaning it's not

107
05:49.280 --> 05:50.160
being triggered

108
05:50.160 --> 05:57.680
by something else like IPSLA, instead it's just acting as a placeholder.

109
05:57.680 --> 06:03.520
We'll also go back into our interface and tell VRRP to track that.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

109
05:57.680 --> 06:03.520
We'll also go back into our interface and tell VRRP to track that.

110
06:03.520 --> 06:10.680
So once again, we'll track tracked object one and decrement our priority by 20.

111
06:10.680 --> 06:12.760
And that does not have IP in front of it.

112
06:12.760 --> 06:17.400
So just VRRP one track one decrement by 20.

113
06:17.400 --> 06:20.880
So now we have a tracked object, which is a stub object.

114
06:20.880 --> 06:24.040
We have VRRP tracking that tracked object.

115
06:24.040 --> 06:31.720
Now all we need is an EEM applet to watch IPSLA and flip the state of this
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

115
06:24.040 --> 06:31.720
Now all we need is an EEM applet to watch IPSLA and flip the state of this

116
06:31.720 --> 06:33.600
tracked object.

117
06:33.600 --> 06:37.000
Now a quick note here as we jump into EEM.

118
06:37.000 --> 06:43.600
This is by no means going to be any kind of an explanation of EEM itself.

119
06:43.600 --> 06:46.840
EEM is a huge topic.

120
06:46.840 --> 06:49.200
It's even bigger than IPSLA.

121
06:49.200 --> 06:52.280
And as you saw, that had a lot of options.

122
06:52.280 --> 06:58.050
EEM is essentially a scripting language that we can use to either write applets

123
06:58.050 --> 06:58.880
directly

124
06:58.880 --> 07:04.840
on the router, or we can also use it to run things like tickle scripts.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

124
06:58.880 --> 07:04.840
on the router, or we can also use it to run things like tickle scripts.

125
07:04.840 --> 07:07.720
And that's again, a whole nother process.

126
07:07.720 --> 07:13.190
We're just going to be looking at some applets that are going to be triggered

127
07:13.190 --> 07:14.120
by IPSLA.

128
07:14.120 --> 07:18.350
So in the interest of time and there's no need to watch me type, I will simply

129
07:18.350 --> 07:18.600
paste

130
07:18.600 --> 07:24.480
in the EEM applets that we're going to use, and then we'll talk about them.

131
07:24.480 --> 07:29.320
So here we have an EEM applet called way and monitor.

132
07:29.320 --> 07:35.240
This applet is the one that's going to make the interface go down when IPSLA
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

132
07:29.320 --> 07:35.240
This applet is the one that's going to make the interface go down when IPSLA

133
07:35.240 --> 07:36.080
triggers

134
07:36.080 --> 07:37.280
it.

135
07:37.280 --> 07:40.320
This of course is our critical line right here.

136
07:40.320 --> 07:46.730
And what this is saying is in the event that IPSLA, operation ID number two,

137
07:46.730 --> 07:47.720
because that's

138
07:47.720 --> 07:55.090
the SLA that we're watching, sends a reaction of jitter source to destination

139
07:55.090 --> 07:56.320
average.

140
07:56.320 --> 07:59.760
Notice we're not putting the numbers in here.

141
07:59.760 --> 08:05.790
We're just saying if IPSLA triggers that event, that trigger is coming from IPS
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

141
07:59.760 --> 08:05.790
We're just saying if IPSLA triggers that event, that trigger is coming from IPS

142
08:05.790 --> 08:07.160
LA itself and

143
08:07.160 --> 08:12.000
the threshold values that we already put in to IPSLA.

144
08:12.000 --> 08:15.720
We're now basically doing two things.

145
08:15.720 --> 08:24.160
First we're manually setting the state of the tracked object one to down.

146
08:24.160 --> 08:30.760
That right there is going to trigger VRRP to now switch over.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

146
08:24.160 --> 08:30.760
That right there is going to trigger VRRP to now switch over.

147
08:30.760 --> 08:33.720
Then we're going to put an output to the console here.

148
08:33.720 --> 08:38.710
We could also, of course, if you wanted to, you could change this to be say a

149
08:38.710 --> 08:40.600
syslog message.

150
08:40.600 --> 08:44.340
That's one of the other things that's available here, but we're going to say

151
08:44.340 --> 08:44.760
just put it to

152
08:44.760 --> 08:46.480
the console for now.

153
08:46.480 --> 08:49.480
We're going to do a WAN failover initiated.

154
08:49.480 --> 08:54.880
Then we have actions 30 through 60, which are simply going in to shut down the

155
08:54.880 --> 08:56.480
WAN interface.

156
08:56.480 --> 09:01.090
Now these last couple steps of shutting down the WAN interface are sort of
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

156
08:56.480 --> 09:01.090
Now these last couple steps of shutting down the WAN interface are sort of

157
09:01.090 --> 09:02.000
optional.

158
09:02.000 --> 09:07.510
The only reason we're doing this is so it still doesn't continuously trigger

159
09:07.510 --> 09:09.520
the IPSLA event.

160
09:09.520 --> 09:14.060
That won't hurt anything because again, we've already changed the state of the

161
09:14.060 --> 09:16.360
tracked object.

162
09:16.360 --> 09:19.160
These last couple of steps are technically optional.

163
09:19.160 --> 09:25.030
It just prevents us from getting essentially it's saying WAN failover initiated

164
09:25.030 --> 09:25.520
over and

165
09:25.520 --> 09:29.480
over and over again every time this triggers.

166
09:29.480 --> 09:36.200
The second script that I have here, the WAN up script is completely optional as
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

166
09:29.480 --> 09:36.200
The second script that I have here, the WAN up script is completely optional as

167
09:36.200 --> 09:36.720
well.

168
09:36.720 --> 09:39.400
Notice the event on this is set to none.

169
09:39.400 --> 09:44.480
This is for us as administrators to manually trigger this.

170
09:44.480 --> 09:49.960
So this is just a quick easy way, as you can see, to fail it back.

171
09:49.960 --> 09:56.160
It's going to set our tracked object back to up causing VRRP to fail back and

172
09:56.160 --> 09:57.200
then steps

173
09:57.200 --> 10:03.160
20 through 50, then have it bring the WAN interface back up.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

173
09:57.200 --> 10:03.160
20 through 50, then have it bring the WAN interface back up.

174
10:03.160 --> 10:09.920
And we again get a console message telling us that the WAN connection is back

175
10:09.920 --> 10:10.040
online.

176
10:10.040 --> 10:13.890
And just to show you what it would look like, since we're still here in this

177
10:13.890 --> 10:14.520
applet, we

178
10:14.520 --> 10:19.400
could do an action where you can actually send a syslog message.

179
10:19.400 --> 10:22.560
And for example, you could set its priority.

180
10:22.560 --> 10:25.920
You could set it to any of the syslog priorities.

181
10:25.920 --> 10:29.800
So maybe this would be a notification.

182
10:29.800 --> 10:35.100
And then you put in MSG for the message and then whatever you want the message
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

182
10:29.800 --> 10:35.100
And then you put in MSG for the message and then whatever you want the message

183
10:35.100 --> 10:35.800
to be.

184
10:35.800 --> 10:41.480
So again, we could just in quotes, put the exact same message in.

185
10:41.480 --> 10:44.560
But that's what a syslog message would look like.

186
10:44.560 --> 10:49.160
This way we can see both the console and the syslog messages.

187
10:49.160 --> 10:56.420
Now another thing, if I were to say do show run section event manager right now

188
10:56.420 --> 10:57.200
, notice

189
10:57.200 --> 11:00.560
the only one that exists is the first one.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

189
10:57.200 --> 11:00.560
the only one that exists is the first one.

190
11:00.560 --> 11:02.920
The second one doesn't exist.

191
11:02.920 --> 11:07.850
EEM applets are only saved when you either exit the applet or just start making

192
11:07.850 --> 11:08.240
a new

193
11:08.240 --> 11:10.000
one, which is what we did.

194
11:10.000 --> 11:15.460
So if we exit and we do the same command again, now you can see we have both

195
11:15.460 --> 11:16.440
scripts.

196
11:16.440 --> 11:20.680
And something else to note, we don't do anything to run these.

197
11:20.680 --> 11:22.320
They're event driven.

198
11:22.320 --> 11:24.240
So these are already active.

199
11:24.240 --> 11:29.300
We can prove that by going right over to our WAN console right now and putting

200
11:29.300 --> 11:29.720
a bad

201
11:29.720 --> 11:32.760
configuration on our WAN.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

201
11:29.720 --> 11:32.760
configuration on our WAN.

202
11:32.760 --> 11:36.800
So here in our port settings, just to really make sure we trigger this, let's

203
11:36.800 --> 11:37.280
make sure

204
11:37.280 --> 11:41.600
we set this to like 500 milliseconds of jitter.

205
11:41.600 --> 11:43.400
That should be plenty.

206
11:43.400 --> 11:45.480
And let's go ahead and apply this.

207
11:45.480 --> 11:47.920
We'll go back over to router one.

208
11:47.920 --> 11:52.560
And here we see everything operated exactly as we want.

209
11:52.560 --> 11:57.800
First, we see that our stub object went from up to down.

210
11:57.800 --> 12:00.480
We then got our EEM message.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

210
11:57.800 --> 12:00.480
We then got our EEM message.

211
12:00.480 --> 12:04.890
Notice it's listed as a log message here that the task WAN monitor is telling

212
12:04.890 --> 12:05.480
us that the

213
12:05.480 --> 12:07.800
WAN failover was initiated.

214
12:07.800 --> 12:11.520
We see OSPF going down because we shut the interface.

215
12:11.520 --> 12:15.840
We see again the console message that gigabit two is administratively down.

216
12:15.840 --> 12:18.560
And of course, then the line protocol goes down.

217
12:18.560 --> 12:25.120
Finally, we see VRRP switchover because we're now from master to backup.

218
12:25.120 --> 12:29.840
If we go back over to our workstation, we do our trace again.

219
12:29.840 --> 12:31.160
And what do we see?
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

219
12:29.840 --> 12:31.160
And what do we see?

220
12:31.160 --> 12:37.150
We're now going through router two to four to the destination exactly as we

221
12:37.150 --> 12:38.280
should be.

222
12:38.280 --> 12:40.960
Let's put our WAN back to normal.

223
12:40.960 --> 12:43.960
So we'll set our jitter back to zero.

224
12:43.960 --> 12:47.880
But of course, as long as we sit here, nothing's going to happen because our W

225
12:47.880 --> 12:48.640
AN interface

226
12:48.640 --> 12:50.120
is down.

227
12:50.120 --> 12:54.200
This is where we have to initiate our failback manually.

228
12:54.200 --> 12:56.960
And that's okay because that was our design.

229
12:56.960 --> 12:58.600
That was our goal.

230
12:58.600 --> 13:04.260
The way you run that script that we wrote manually is you say event manager run
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

230
12:58.600 --> 13:04.260
The way you run that script that we wrote manually is you say event manager run

231
13:04.260 --> 13:04.560
and then

232
13:04.560 --> 13:08.600
you put in the name, which of course we had named it WAN up.

233
13:08.600 --> 13:10.320
So let's hit enter.

234
13:10.320 --> 13:15.680
So as our script runs, the first thing we see here is our syslog message.

235
13:15.680 --> 13:18.600
Then of course, we see the stub object come back up.

236
13:18.600 --> 13:23.280
We see our log message come in, which is of course the console message.

237
13:23.280 --> 13:28.660
Then we switch from backup to master as VRRP takes back over and our interfaces

238
13:28.660 --> 13:29.400
come back

239
13:29.400 --> 13:32.640
up as well as OSPF.
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

239
13:29.400 --> 13:32.640
up as well as OSPF.

240
13:32.640 --> 13:37.480
And now everything is failback as it was when we began.

241
13:37.480 --> 13:44.470
So in this video, we saw how to use EEM to have this only failover once and

242
13:44.470 --> 13:45.200
then stay

243
13:45.200 --> 13:51.070
down until we fix the WAN, make sure it's working and then manually fail it

244
13:51.070 --> 13:51.840
back.

245
13:51.840 --> 13:55.400
And of course, we wrote another script to do the manual failback.

246
13:55.400 --> 13:57.080
You of course don't have to do that.

247
13:57.080 --> 13:59.920
You could always just do it manually.

248
13:59.920 --> 14:04.630
Next up, let's take a look at some load balancing options that we have instead
WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

248
13:59.920 --> 14:04.630
Next up, let's take a look at some load balancing options that we have instead

249
14:04.630 --> 14:05.760
of just doing

250
14:05.760 --> 14:07.280
Fail over and fail back.

251
14:07.280 --> 14:29.640
[ Silence ]