You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DOCS-2997: Write the guide for preparing an existing bridge
Fill in the guide for nodes whose only uplink also carries the node's own IP
address, where Calico cannot create the bridge itself.
The guide is written around configuring the host to bring the bridge up
correctly at boot. Moving a running IP address onto a bridge is the dangerous
way to reach the same end state, so the page does not teach it.
The page states the three properties Calico checks, and what it deliberately
does not require. It covers why the bridge MAC address must be set explicitly
and must not be the trunk's, the VLAN membership the bridge device needs in its
own right, and two hazards that are not settings: a bridge named with the
prefix Calico treats as its own, and the startup cost of spanning tree.
Verification runs in two rounds, because the failures that matter most only
appear when the first workload attaches.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@@ -6,66 +6,308 @@ description: Configure a Linux bridge on your nodes so Calico Enterprise can use
6
6
7
7
:::note
8
8
9
-
L2 bridge networking is a tech preview feature. APIs and behavior may change before GA.
9
+
L2 bridge networking is a tech preview feature.
10
+
APIs and behavior may change before GA.
10
11
11
12
:::
12
13
13
-
Use this guide when the uplink that must carry your VLANs is also the one your node's IP address depends on. $[prodname] cannot take that interface over on its own, so you build the bridge and $[prodname] uses it.
14
+
Use this guide when the interface that must carry your VLANs is also the one your node's IP address depends on.
15
+
$[prodname] cannot take that interface over on its own, so you build the bridge and $[prodname] uses it.
16
+
For why, see [Who owns the host network configuration](about-l2-bridge.mdx#who-owns-the-host-network-configuration).
14
17
15
-
The goal is not to move a running address onto a bridge. It is to have the host bring the bridge up correctly at boot, so the node's address arrives in the right place the first time.
18
+
If $[prodname] can have an interface of its own, you do not need this guide.
19
+
Use [Connect workloads to an existing VLAN](connect-vlan.mdx) instead.
20
+
21
+
:::note
22
+
23
+
The goal is not to move a running IP address onto a bridge.
24
+
It is to have the host bring the bridge up correctly at boot, so the node's address arrives on the right device the first time.
25
+
Configure this the way you configure the rest of your host networking, and verify it survives a reboot.
26
+
27
+
:::
16
28
17
29
## Before you begin
18
30
19
-
This procedure changes how your nodes get their IP addresses. Read this section before you start.
31
+
This procedure changes how your nodes obtain their IP addresses.
32
+
A mistake takes the node off the network, and you will not be able to fix it over the network.
33
+
34
+
You need:
35
+
36
+
- Root access to the node, and ownership of its network configuration.
37
+
- Out-of-band access to the node, such as a console, or a tested way to roll the configuration back.
38
+
- The name of the interface that will become the trunk.
39
+
- The VLAN your host's own traffic uses, or confirmation that it is untagged.
40
+
- The VLAN IDs your workloads will use.
41
+
- The node's IP address, prefix length, and default gateway.
42
+
43
+
Work on one non-critical node first, confirm it, and only then apply the same configuration to the rest of the cluster.
20
44
21
45
## What you are building
22
46
23
-
Before and after, so the configuration further down makes sense.
47
+
Today, your node's IP address sits directly on a physical interface, and that interface carries the node's traffic.
48
+
49
+
Afterwards, that interface is a port on a bridge and carries no address of its own.
50
+
The bridge carries your workload VLANs, and the node's address sits either on the bridge or on a VLAN device layered on top of it.
51
+
$[prodname] attaches workloads to the same bridge as access ports.
24
52
25
53
## What $[prodname] requires
26
54
27
-
Three settings on the bridge. $[prodname] checks all three and will not use a bridge that is missing any of them.
55
+
$[prodname] checks three properties of the bridge and refuses to use it if any are missing.
56
+
It does not set them for you, because your host network configuration owns them and would undo the change.
57
+
58
+
| Requirement | Setting | Why |
59
+
| ----------- | ------- | --- |
60
+
| VLAN filtering enabled |`vlan_filtering=1`| Without it the bridge ignores VLAN tags entirely, so segments are not isolated from each other. |
61
+
| An explicitly set MAC address | Any address, set deliberately | Stops the bridge's address changing as workloads come and go. See below. |
62
+
| The 802.1Q tag protocol |`vlan_protocol=802.1Q`| $[prodname] programs 802.1Q VLAN membership and creates 802.1Q sub-devices. On an 802.1ad bridge none of it matches. |
63
+
64
+
:::caution
65
+
66
+
Enable VLAN filtering **before** you put the node's address on the bridge, or as part of the same configuration change.
67
+
Turning it on makes the kernel start enforcing VLAN membership immediately, and an address on a bridge with no VLAN membership stops receiving and sending traffic the moment filtering comes on.
68
+
69
+
:::
28
70
29
71
### Why the MAC address must be pinned, and must not be the trunk's
30
72
31
-
Two separate failures, both avoided by the same setting.
73
+
A Linux bridge has no MAC address of its own.
74
+
By default the kernel borrows one from whichever attached port has the numerically lowest address, and it re-runs that choice every time a port is added or removed.
75
+
76
+
On an L2 bridge the ports are workload interfaces, which appear and disappear constantly.
77
+
Left to itself, the bridge's address changes under your node, and if the node's IP address is on that bridge, its traffic stops being delivered.
78
+
79
+
Setting an address explicitly stops the kernel from ever reconsidering it.
80
+
81
+
Do not use the trunk interface's own MAC address for the bridge.
82
+
When two devices in the same path present the same address, some network card drivers reject the second programming attempt and the bridge is left non-functional.
83
+
Use a distinct address.
84
+
85
+
A locally administered address is a good choice, because it cannot collide with a real card.
86
+
Set the second least significant bit of the first byte: `02:00:00:00:00:01` and `0a:1b:2c:3d:4e:5f` are both locally administered.
32
87
33
88
### What $[prodname] does not require
34
89
35
-
Just as useful as the required list, and it keeps the required list credible.
90
+
Just as useful as the list above, and it keeps that list credible.
91
+
92
+
-**Enslaving the trunk.**
93
+
$[prodname] adds the trunk interface to the bridge itself.
94
+
The interface only has to exist and be named in the `Network`.
95
+
-**A particular default PVID.**
96
+
$[prodname] does not read or change the bridge's default PVID.
97
+
-**Removing an existing VLAN device.**
98
+
If you already have a VLAN sub-device on the bridge, $[prodname] reuses it rather than failing.
99
+
-**Bringing the bridge up.**
100
+
$[prodname] brings up a bridge that is administratively down.
101
+
-**Self VLAN memberships.**
102
+
$[prodname] never adds or removes the bridge device's own VLAN memberships, or their PVID and untagged flags.
103
+
They are yours to manage.
36
104
37
105
### Two hazards that are not settings
38
106
39
-
A naming collision that deletes your bridge, and an option that quietly slows workload startup.
107
+
**Do not name the bridge with a `calb-` prefix.**
108
+
$[prodname] treats that prefix as meaning it owns the bridge, and it deletes bridges it owns when they are no longer needed.
109
+
A bridge of your own named that way can be deleted from under you.
110
+
Nothing rejects the name when you create the `Network`, so this is on you to avoid.
111
+
112
+
**Spanning Tree Protocol is allowed, but it costs workload startup time.**
113
+
Every new access port waits out a forwarding delay before it passes traffic, which delays every workload attaching to the bridge.
114
+
$[prodname] warns once and leaves STP alone on a bridge it did not create.
115
+
Turn it off unless you need it.
40
116
41
117
## Configure the host
42
118
43
-
Write this as persistent configuration. A change that does not survive a reboot is a trap.
119
+
Write this as persistent configuration, using whatever manages networking on your nodes.
120
+
A change made with `ip` commands alone disappears at the next reboot, and the node comes back with its address on an interface that is now a bridge port.
121
+
122
+
The steps below describe what the configuration has to achieve.
123
+
The example that follows shows one way to express it.
124
+
125
+
1. Create a bridge, with VLAN filtering enabled, the 802.1Q tag protocol, STP disabled, and an explicitly set MAC address that is not the trunk's.
126
+
2. Add the trunk interface to the bridge as a port, and give that port membership of every VLAN it must carry, including the VLAN your host traffic uses.
127
+
3. Give the bridge device itself membership of the VLAN your host traffic uses.
128
+
See [Self VLAN membership](#self-vlan-membership).
129
+
4. Remove the IP configuration from the physical interface, so it carries no address.
130
+
5. Put the node's IP address, prefix, and default gateway on the bridge or on a VLAN device above it, depending on whether your host traffic is tagged.
131
+
See [Where the host's address goes](#where-the-hosts-address-goes).
132
+
6. Apply the configuration and confirm the node is still reachable.
133
+
7. Reboot the node and confirm it comes back with the same addressing.
44
134
45
135
### Where the host's address goes
46
136
47
-
Two shapes, depending on whether your host's own traffic is tagged.
137
+
Two shapes, depending on your host's own traffic.
138
+
139
+
**Untagged host traffic.**
140
+
Put the node's address directly on the bridge device.
141
+
The bridge port facing the trunk needs the host's traffic to arrive untagged, which means setting a native VLAN on the `Network` later so $[prodname] knows which VLAN untagged traffic belongs to.
142
+
143
+
**Host traffic on its own VLAN.**
144
+
Create a VLAN device on top of the bridge for that VLAN — for example, VLAN 100 on `br-l2` gives `br-l2.100` — and put the node's address on the VLAN device.
145
+
The bridge itself carries no address.
146
+
147
+
The distinction matters and is easy to invert.
148
+
A VLAN device **above** the bridge, holding the host's address, is the supported arrangement.
149
+
A VLAN device **below** the bridge, standing in for the trunk, is not supported.
48
150
49
151
### Self VLAN membership
50
152
51
-
The step that is easiest to miss, and the one most likely to take a node off the network.
153
+
For the host to receive traffic on a VLAN through the bridge, the bridge device needs membership of that VLAN in its own right.
154
+
This is separate from the membership you give the trunk port, and it is the step most likely to be missed.
155
+
156
+
Without it, the bridge forwards that VLAN's traffic between its ports but never delivers any of it to the host, so a node whose address depends on that VLAN goes dark.
157
+
158
+
$[prodname] does not add this membership and does not remove one you have added.
159
+
160
+
Some configuration tools have no field for it.
161
+
Where that is the case, set it with a hook that runs when the bridge comes up:
162
+
163
+
```bash
164
+
bridge vlan add dev br-l2 vid 100 self
165
+
```
166
+
167
+
Confirm it with `bridge vlan show`, where the bridge device appears in the list alongside its ports.
52
168
53
169
### An annotated example
54
170
55
-
A complete configuration with every line explained.
171
+
This example uses NetworkManager keyfiles, for a node whose management traffic is on VLAN 100 and whose workloads will use VLANs 10 and 20.
[ "$1"="br-l2" ] && [ "$2"="up" ] && bridge vlan add dev br-l2 vid 100 self
238
+
exit 0
239
+
```
240
+
241
+
Make it executable and owned by root, or NetworkManager ignores it.
242
+
243
+
:::caution
244
+
245
+
Validate this in a lab before you apply it to a cluster you care about.
246
+
Host network configuration varies with distribution and with whatever manages it, and the failure mode is a node you cannot reach.
247
+
248
+
:::
56
249
57
250
## Point Felix at the right L3 device
58
251
59
-
Once the node's address lives on the bridge, $[prodname] needs to be told where to find it.
252
+
Once the node's address is on the bridge or on a VLAN device above it, $[prodname] has to be told where to find it.
253
+
Otherwise it attaches its programs to the physical interface, which no longer has an address, and traffic between workloads on different nodes fails.
254
+
255
+
1. Find the current value of `bpfDataIfacePattern` in the default [FelixConfiguration](../../reference/resources/felixconfig.mdx).
256
+
2. Add a pattern that matches the device holding the node's address.
257
+
For a node whose address is on `br-l2.100`, `br-l2.*` matches it.
258
+
3. Apply the change and confirm `calico-node` restarts cleanly.
259
+
260
+
You do not need to add the trunk interface to this pattern.
261
+
$[prodname] attaches to the trunk as part of setting up the L2 network.
60
262
61
263
## Verify before handing the bridge over
62
264
63
-
Two rounds of checks. Some failures only appear when the first workload attaches.
265
+
Check in two rounds.
266
+
Some problems only appear when the first workload attaches, so the first round is not sufficient on its own.
267
+
268
+
Before you create a `Network`:
269
+
270
+
1. Confirm the node is reachable on its own address, and that its default route is intact.
271
+
2. Run `ip -br addr show` and confirm the address is on the device you intended and the physical interface has none.
272
+
3. Run `bridge link show` and confirm the trunk is a port of the bridge.
273
+
4. Run `bridge vlan show` and confirm the trunk port carries every VLAN you expect, and that the bridge device itself appears with the host VLAN.
274
+
5. Reboot the node.
275
+
Repeat steps 1 to 4.
276
+
277
+
After you create a `Network` and attach the first workload:
278
+
279
+
1. Confirm the node is still reachable.
280
+
A bridge MAC that was not pinned changes at this point.
281
+
2. Run `bridge link show` again and confirm the workload's interface joined the bridge.
282
+
3. Confirm the workload can reach its gateway.
64
283
65
284
## Reference the bridge from a Network
66
285
286
+
1. Set the bridge in the `Network`'s host configuration:
287
+
288
+
```yaml
289
+
bridge:
290
+
existingBridge:
291
+
name: br-l2
292
+
```
293
+
294
+
2. Name the trunk interface in the same host configuration entry.
295
+
Omitting it leaves the node unable to route workload traffic.
296
+
3. Continue with [Connect workloads to an existing VLAN](connect-vlan.mdx) to create the IP pools, the attachment definitions, and your first workload.
297
+
67
298
## If $[prodname] will not use your bridge
68
299
69
-
What the refusal looks like, and where to look first.
300
+
$[prodname] fails closed.
301
+
If the bridge does not meet all three requirements, nothing is programmed and workloads on that `Network` stay administratively down rather than attaching to a bridge that cannot isolate them.
302
+
303
+
The only signal is the Felix log in `calico-node`, which names the bridge, the `Network`, and the command that fixes it, and repeats every few seconds until you do.
304
+
There is no Kubernetes event and no status on the workload.
305
+
306
+
See [Troubleshoot L2 network connectivity](troubleshoot.mdx).
0 commit comments