hey good morning everybody. thank you for coming along. 9am, it's exciting tobe here in atlanta. my name is paul andrew, i'm a technical product manageron the office 365 team. and i cover networking, datacenter performance for office 365. and i also work on someof our new features that we're looking at forlarge multi national companies. and i'm joined by paul collinge.
>> hi everyone, i'm paul collinge. i'm a senior program managerwithin our office 365 px team and i spend the vast majority of mytime working with customers, helping them connectto our services and solving problems around the networkand connectivity space. >> so we're here today to talk toyou all about cloud networking. what does it mean toconnect to a saas service? and it really only mattersif you have lots of users, if you're not just a single user athome connecting to the internet,
that's all gonna work, right? that's all easy. we're here to talk to you ifyou have a thousand users, all doing it at the same time,across the same network connection, and what does that mean? so, that's the spacethat we look at, and we actually have two sessions today. how many people are joining us forboth sessions and already know about that?
a couple there. i thought we'd talk just a littlebit about how we broken these two sessions up. and the first session is, what we wanna talk through asa step by step, how to go about implementing expressroute on aparticular customer implementation. so, there's a few steps that wewanna think about going through before you switch thosenetwork routes over to using expressroutefrom using internet.
so that's what we'regonna talk through on the first session this morning. and so we have ten, we've brokenthese up into ten different areas and we'll talk througheach one individually starting at what you do first, goingall the way through to after you've switched over if you have todo some trouble shooting. so that will be the first hour orso that we'll be talking. the next session that we have,which is at 10:45, it's over in building b.
that session is more sortof discussion topics around network routing, specificallyrelated to expressroute. so that's the session that we'llgo into things like, well, when should you be considering usingexpressroute direct networking versus just using the internet,or some combination of the two? and by the way, spoiler alert, ifyou're gonna be using expressroute you are using some combinationof expressroute internet. there is no expressrouteonly option for office 365. we'll talk about different waysthat you can do routing from your
user pcs, through your land to thepermanent network to get that split between expressroute networking andinternet networking. we'll generally just try to sharewith you some of the things that we've learned over the year or sothat expressroute has been out but people have gone throughto do that implementation. so that's gonna beour second session. we'll also talk a little bit about,in that session some of the process that has evolvedaround expressroute, around review of new customersthat wanna use expressroute.
we do have a process you have to gothrough to have the requirements of your network reviewedbefore we will say okay, it's a good idea, you shouldgo ahead and use expressroute. and we have that review process sothat we can help people to make the right choices about networkingoptions that they have. so we'll talk about thatin that session as well. okay, so, this session we've gotten steps that we're gonna go through and these ten steps largelyassume that you're ready to go, you're ready to set upexpressroute in an organization.
you may have one ormore locations with users. you may have a wan acrossmultiple buildings, you might just have a lan. we wanna think about what are allthe things that you have to do in order to make sure that you havea successful network update? so we're gonna go through,and we'll switch back and forward a little bit, andwe'll probably interrupt each other. >> undoubtedly. >> and add a little bit.
we've been working together for a little while nowon cloud networking. so we're gonna talk to you aboutpre-requisites before you get going, before you start oneof these projects. what kind of requirements you wannagather from your organization that you're working with? it's never a case that therequirement is just to deploy direct networking. there are alwaysnetworking requirements
that you have to becognizant of as well. we'll talk about a project plan thatyou might use to create this kind of project. how you would go about doinga network assessment and different types of networkassessments that you might use. how to think about network securitywhere it's not just internet network security, it's not just yourlan network security, but it's something else. it's a direct network connectionbetween your network and
microsoft's data center networks. so it's not really either of those, it's kind of somethingin the middle. we'll talk about an implementationplan to prepare, then perimeter network routing design,what goes in your perimeter network. lan specific designs, which is connection from the usersto that perimeter network and then test and deployment plans, switchingthe routing on and trouble shooting. so that's what we've got,let's get going.
all right. so first up, why? why do we have to havean implementation project for adding direct networking? >> you might find that list we wentthrough is quite a lot, because you may be already using expressroutewith azure private peering. and it was pretty easy to turn on,you put the circuit in, you turn it up, you connectyour azure private devices. that's pretty easy, relatively easy.
so it kinda looks like this. the changes are required oncethat circuit's in place, your chosen providersput that link in, it's connected to microsoft'snetwork wherever you've chosen. you can turn that circuit up andyou can either connect to your constrained vnet there in azure oryou can't. it either works or it doesn't. if it doesn't work thenwe can work through and fix that problem with the circuit.
so there's relatively little changethat needs to be done within your environment once thatcircuit's plugged in. but if you then move on to think,okay, microsoft peering, which is thepeering model we use for office 365, we wanna turn that up and givethat a go and see how that works. and this is where we get intodifficulty i think because it's so easy for the azure private pairing,that it's easy to presume that the microsoft pairingis as easy and it is not. so, when you turn on microsoftpairing, all those red dots
indicate somewhere that you needto be thinking about changes and management. so that could be firewalls,proxies, routing tables, security devices, nat devices,the whole array of things that need to be thought ofbefore you turn that circuit on. because if you don'tmanage those things before you turn microsoft pairing on,there's a very, very high chance of a complete outageof office 365 in your environment. and, obviously,nobody wants that to occur.
>> yeah, so why is there all thesechanges required with microsoft peering where it's not requiredwith the age of private peering? what are we thinking about there? >> well, the majority of the workneeds to be done because we're accessing public ip space. and when you open thatsecond circuit to microsoft's infrastructure, it's essentially routinga override to the internet path. so once you turn that on you havetwo possible routes to the same
end point. the end points you connectto via expressroute are exactly the same thatare published over the internet. and the networking guyshere we think we've got possible asymmetric routing problemwhich we'll talk about later. and that's a very very real risk. the traffic will come in one wayonce you've turn that circuit on and come out of another. so the whole raft of thingsthat need to be considered and
to work through this project youwould think with azure peering, you probably just need your networkteam to get that circuit spun up. when it's ready, you go,okay, put it live, and we'll connect to our machines. but when we look at office 365, that network team are obviouslyinvolved at that level, but then we need to start engaging otherteams to ensure that every piece where that red dot was shown on theprevious diagram is thought about, mapped out, and we ensure thatwe're connecting in the right way.
so that would be if you're usingproxies for internet access. there's work needs to be donethere to adjust the traffic, the allow list through that proxy, to ensure that proxy can scaleup to code with the traffic. again, this is not an exhaustivelist of what needs to be done here. i'm just trying to give you an ideaof the teams you need to pull in to ensure success in this area. so on top of the proxy team, you'llneed your desktop team engagement. how are you gonna managethat traffic on a client?
are you gonna use multiple proxies? we talked about your routes andmethods in the next session but this will involved the desktop team orsoftware versions you're gonna use. to connect in the best way, how are you gonna manage thatpac file on the client to distribute that routing as theinternet and express route model? so that needs to be considered. the office 365 project team willneed to be involved obviously, to plan through this,pull the talk through in a kind of
typical implementation plan, whichis normally led by the project team. they may need to configurethings in the service and to set up the service towork how you want it. maybe now you've got the extrabandwidth you wanna change how skype for business works,the codecs you use etc. policy setting on top of the projectteam your network architects are gonna not just think of thatcircuit that we're putting in or multiple express route circuits. we've got to look atthe whole network.
how are we gonna get that trafficfrom that client to the correct ingress? normally we recommend multipleexpress route circuits for failover reasons. and for a multinational that'sabsolutely gonna be necessary. or do we use the internet model forsome sites. and your network architectsneeds to look at where it makes sense to usethat expressroute circuit. where we need expressroute circuits?
where we needthe internet improving? all this work needs to be donein advance of turning those circuits up. exchange, so some of the complications we seearound hybrid environments, and so that might be sharepoint, skype forbusiness hybrid, or exchange hybrid. and we'll talk a bit aboutexchange hybrid in the next session. around what happens whenyou put expressroute and you've got exchange hybrid.
because some connections need tocome in to those services from our online services. so how'd you manage that? now you have twopossible ingress points. do you want them to comein over the internet or do you want to use expressroute? if you use expressroute, what's gonna happen to your usersthat are in starbucks somewhere? so these things needto be considered.
and then, last but not least,your firewall security team. i'll leave it to you to decide whoargues with who on that bottom one, but it's a pretty common thing thati get involved with with customers. where the project team want to dosomething and the security team are slamming their hand down,and quite often quite rightly. so these discussions need to be done well in advance of anycircuit going live. so ensure that circuit is securedbecause one of the common misconceptions with expressroute for
office 365 is that it givesyou a security barrier. you're connecting topublic infrastructure, you're connecting tomicrosoft's infrastructure and that still needs securing andthinking about in advance, okay? >> all right solet's talk about prerequisites. and in prerequisites what we're really talking about is thereis somethings you have to have before you start thisimplementation project. and this implementation projectwe're talking about is really about
your routing updates thatyou're gonna be making when you have direct networking. paul c talked about you having a route both acrossan internet connection and across express route essentially to thesame set of servers at microsoft. now, that's relativelystraight forward if you have advertisedroutes to connect to those. but the thing is with routes, routesare advertised from the server end and you will connectfrom the client and so
you'll see those routes as a client. and you'll say okayi go this way and so as a client it's veryeasy to see those routes and you'll get the priority and yourcomputer makes a great choice based on that priority to go either overexpress routes or over the internet. it's all very straight forward,very simple. but those routes had tohappen both ways and those routes have to bethe same in both directions. so the routes are advertised bymicrosoft both over the internet and
over expressroute. you'll get a higher priorityroute over expressroute. and microsoft takescare of that routing, the trick is that onthe customer's side, on the customer lan, you've also gotto advertise routes to microsoft. why do you have to do that? there's two reasons. one is because there's gotta bea return packet for every request. there's gotta be a response.
so if you seen a request toa microsoft server and say, i don't know, get me my email. there's got to be a responseto that tcp packet and it's got to go over the same path. the reason it has to go overthe same path is because if your response goes over a different path,did we mention asymmetric routing? if your response goesover a different path, we have firewalls that are gonnago i didn't expect this response because there was no request,drop the packet.
and that simply means your requestfor that email is going to fail. and as your user is sittingthere running outlook, they will just seethat they get no email. so now we have these two paths. the other reason thatyou have to have those two paths set up correctly withroute advertisements from microsoft, route advertisements from thecustomer, is because microsoft, and paul mentioned this as well, makessome requests in to the customer. whatever those requests are ithink we talk mostly about this in
the next session. but there are things like hybridexchange, hybrid sharepoints, adfs. there's a variety of differentthings that a customer or anyone might have set up on-premisisthat microsoft connects to. same thing has to happen. the microsoft server has to connectto the on-premisis server and then the response has togo back the same way. so pre-requisites forall of that and we're gonna be talking mostlythrough how you get that right.
how you make sure that routing isset up and so that you have a good experience when you switch it over,and everything still works. before we get there,we have some prerequisites. and so, what are they? i've mentioned that there is areview process that we go through to make sure that there isa good use of express route. we really don't want people havingasymmetric graphing errors and failing to connect. so we wanna make sure that peopleappreciate that that's gonna happen.
we unfortunately get a lot ofassumptions that this could just be turned on, it's gonna work andeverything's taken care of and that routing change means that itis not, you have to do some work. and so there is a reviewprocess today that you have to work with a microsoftaccount manager. we'll talk in detail aboutthat at the next section. you've got to have an expressrouteservice provider, the expressroute circuit from microsoft terminatesat a network service provider. and so the customer thatyou're working with or
yourself if you're the customer youhave to have a network operator that is a microsoft expressroutepartner. and they run the cablesfrom the lan or the permanent network that you have. into the co-locationfacility with microsoft. so there is the customer network,there's the network operator, the one network operatoryou work with and then there is microsoft's network. >> and it's worth pointing outhere that that expressroute
circuit terminates a pair pointwith microsoft's global network. it does not go into the data center,which is a common misconception i run into where customers think abouttheir data in one location and want to run a circuitinto that location. that's not how this works,how the internet works. the isp or your expressrouteprovider will run a circuit to a an internet exchange andhand it to microsoft's network and then we get it to where it's goingto go on our network from there. >> so you got to have one ofthose network service providers.
you should read our documentationboth on azure expressroute, which is the networkingtechnology that we work with. paul andi both work in the office 365 team. it's another great team in azurenetworking that actually do all of the software defined networkingtechnology that underlies this. we rely on them very heavily. but they do have a separateset of documentation which you wanna read and understand. so we have documentation on theoffice 365 side which mostly focuses
on the higher layers about networkconnectivity for our services. and it's super helpful if youhave some good base level understanding of tcp/ip. if you've done work with tcp/ipany time in the last 25 years, that stuff's allpretty much the same. if you haven't, learn about it. find a good book andlearn about some of these protocols. it's really valuable to havea little bit of understanding about some of these routing protocols.
so that you're not just relyingon me waving hands of servers and clients and that sort of thing. it is a good to get a bit ofthat background knowledge. so that is our prerequisites,what is up next? we're gonna gather requirements. so we wanna gather requirementsbecause networking is not always the same. and something we've found as we'vebeen working with a lot of customers connecting into microsoft'scloud is that most networks,
most lans are different. there are differencesthat make it a lot of time spent asking people abouthow they've configured their lan. how distributed it is? what kind of routing they have? how their pcs connect? there are a lot ofdifferences there. and sogathering requirements is about learning about those differences,
learning about what peopleare involved in the organization. paul had this great slideshowing the different teams that are involved. it's very common for us to getinvolved with one of those teams and then have to spend some timefinding either the team that owns the routers orthe team that owns the firewalls. we're always talking with largeorganizations where you're talking about dedicated networking andyou have to make sure you have the right people involved, otherwisethose projects go pretty slowly.
so, another thing i've got listedhere is cataloging outbound and inbound services. now, an outbound serviceis just really a user and their client running outlook orrunning onedrive for business sync tool orrunning skype for business and making a tcp connection out intomicrosoft's data centers somewhere. so, that would bean outbound service. and these outbound services normallygo through a proxy server that you have, maybe a firewallthat you have hopefully
before they connectonto the internet. and they're relativelywell understood. and typically a lot of themgo through the same paths. maybe there's other devices inthere which are intermediary. maybe there's devices dependingon what kind of protocol it is. but they're fairly standacross most customers. the inbound services are usuallypretty different and inbound services are alloptional for office 365. you don't have to haveany inbound services.
so sometimes we'll startworking with a customer and they'll be asking methe microsoft guy, what inbound services dowe have on our network? i don't know what ports didyou open on your firewall for inbound services? so inbound services the peoplewho set up the lan and set up the firewallconnecting into that lan should know what thoseinbound services are. irrespective of how you get there,we need to know what they all
are because we're gonna makea significant change to the routing for this network andeach of those inbound services needs to have a route advertisedto microsoft correctly. both over the express route linksand over the internet links so that windows inbound servicesare being used by microsoft servers we will send requests overthe right network, so. >> andthis is why you need multiple teams. because your firewall teams maybehave been asked open this port. and they don't ask why.
>> they normally do.>> but don't ask why. >> [laugh] they just do it, right. >> and then argue a bit, argue somemore, and then open the ports. but they might not knowthe exact reason why. so you need your exchangehybrid engineer to explain, okay, we're exposing owa, autodiscover, etc., to the internet. and we need to understand that. so that's why you need to pullthose teams together to get a full end-to-end picture,to understand, okay,
how's this gonna look when we put asection route out to these devices? how are we gonna connect in and out? >> all right, sonext thing we wanna know. and we're gonna talk more ina second about inbound services. next thing we want to know is whatare the security requirements that you have on the network? now, we're gonna diveinto security and how you address thoserequirements in a minute but the point about gatheringrequirements is we wanna know what
kind of level of securityis required on the network. same thing for high availability. are you concerned if there'sa single point of failure on the network at some location,or are you not? expressroute does a great job ofremoving single points of failure. when you provision expressroute, we have an availability sla forthe microsoft components. many network providers who workwith expressroute also have those availability guarantees and
for microsoft, it's a refundbased backed guarantee, so we'll give you money backif we don't meet that sla. we achieved that sla, by the way, while having no singlepoints of failure. we run multiple physical cables, we had multiple routersat each location. we run double the amount ofbandwidth that the customers are paying for, so if we havea physical outage of some kind we're still able to deliverthe bandwidth that has been paid for
by the customer. and so that's how we backthat availability guarantee. but what are the highavailability requirements? do you need expressrouteconnectivity in multiple locations in order to achieve the availabilityrequirements that you have? do you need to have multiple inthe same location to do that. >> i'll give you an example ofsomething that happened a while back a few months ago in the uk thati won't name the provider, but a sensor went off line anexchange sensor that went off line
because of a power failure,only for seven minutes, but if you circuit is running therewe have no control over that. that's not our internet exchange. and it didn't take microsoft onlinebecause it was just a section that we're not in. but you've got to think about that. what is your plan if you havea single expressroute circuit and that provider's buildinghas some sort of incident. do we have a secondary circuit wherewe can switch that traffic over very
quickly? or have we got policy to switchthat traffic over to the internet until that circuit is back up? so these need thinkingout well in advance so we can plan those extra circuits,the bandwidth, etcetera. >> all right, so here is a list ofthe inbound traffic types that we most commonly see. so adfs, active directoryfederation services. various microsoft servers, exchangeserver, azure ad, connects into adfs
on premises servers in order to dopassword or security validation for users signing in ifyou're using any of this. exchange server hybrid inbound email from exchange on line to maybe anon-premises email server if you have an actp mx in pointthat's on-premises. share point online mail,if you have on-premises email again, then you've got a mail hoston premises accepting that. we've got sharepoint federatedhybrid search, if you are using that or if you are using businessconnectivity services, skype for
business hybrid, skype forbusiness federation, both would have servers on-premises in the customerorganization for being connected to. and then skype forbusiness cloud connector. not one i'm familiar with,but it also involves a server on-premises that will beconnected to from outside. so all those inbound flows, theyare the high risk ones typically for this asymmetric routing system. and they're typical higher risk,because you've got to actively make a routing change, andadvertise routes from on premises to
the outside and those routeupdates are pretty different for teams that are familiar withhaving just internet connectivity. if you just havean internet connection, your routes are prettystraightforward. you're advertising a route for everything that you have on-premisesout that one internet connection. as soon as you have thisseparate dedicated route, your advertisements geta lot more complicated. all right,another prerequisite we wanna do
is get hold of networktopology diagrams. a topology diagram is reallycritical to be able to figure out, how am i gonna setup this routing? what are my ip address that i'mdealing with and therefore how am i gonna do either my bgp or myother type of route advertisements? so when we get involved in theseprojects we wanna see these topology diagrams and understand really what the routingis happening on the network. so here's an example ofone we put together.
we published this diagramon sport.office.com. by the way, we have an articlethat talks you through all of the same stuff that we're talkingabout, this implementation plan. so that's online andthis is really helpful for someone doing troubleshootingof the routing or for planning of that routing tohave this kind of diagrams. >> and it's also where going throughyour inbound traffic flows that we just talked about andmapping those onto a diagram. so you could either providethat to someone else when,
if you need support or just for your own reference so the wholeteam can look at okay, and then understand how that traffic flowsin and out of your environment. all right, next up, meet-me orcall location facilities, meet-me locations. you've got to figure out where youwant to have your expressroute connectivity go to tomeet with microsoft. we have a few dozen of theselocations around the world today where expressrouteconnectivity comes in.
by the way if you're thinking aboutan expressroute direct networking versus all internet orversus some internet only and some expressroute plus internet. we have two or three times thenumber of co-locations facilities were we have internet connectivity. >> 65 [crosstalk] icounted the other day. >> 65, so 65 locations aroundthe world where we have internet connectivity frommicrosoft's global network. we run the cold potato routinginstead of hot potato, so
we will attract traffic ontomicrosoft's global network if it is connecting to our servers. why do we do that? because we want you guys tohave a great experience and have good performance and we puta lot of money into our network, a lot of excess capacity. so we want to advertiseroutes to get that as quickly as possibleonto our network. if you're internet connected,65 locations where we can do that.
expressroute you have to choose. and if you're going expressroute asmuch as possible, you wanna have as many as possible locations whereyou connect in so you get into that microsoft network and let us routeyour traffic around the world. you can literally have anexpressroute connection in australia and we'll route your traffic fromaustralia to wherever the data center is that you need toconnect to for microsoft. and if you have users in europeyou can connect onto our network in europe and we'll route yourtraffic across that as well.
you don't have to traverse theinternet with your your traffic or have connectivityacross the world to where the microsoft datacenter is that you have. and i talked about this a littlebit yesterday on my talk around trying to reset kind ofthoughts around the internet. and the service is designed towork over the internet primarily. and we have those network peerpoints all over the globe. so when you think about traffictraveling from the edge of your network to ours,that's the internet leg.
and hopefully that's very short,i did a demo yesterday from here. there were six hops from hereto the microsoft pair point, which happens to be in atlanta. so comcast is the isp here,we watched the trace go out, and paired with microsoft'snetwork in one millisecond. so, that is the sort of the thingthat we need to think about in terms of the internet, that we're only onthat very short leg hopefully to where you meet microsoft's networkand then we'll take it from there. an expressroute is purely a privateconnection to that pair point or
one of those pair points. so there you go, the first pairing point wehave there listed is atlanta. we have expressroute pairing and internet pairingsomewhere around here. all right we will move on, so numberthree, actually we're at three and we're 30 minutes in i notice. so number three is a project plan. we encourage you tohave a project plan for
doing these routing changes. i got an example herein microsoft project. network assessments, selectingthe expressroute provider, learning about the configuration, gatheringrequirements, planning outbound services, inbound services, settingup a testplan, a deployment plan. deploying during an outage,that's always interesting. when you're making a routing changevery often you're doing it on the production network. and so by definition,
you've gotta have a networkoutage in order to do that. and then in order to testthat everything's working and potentially roll back, so we'llcome back to that in a little bit. but let's move on tonetwork assessments. >> turn my microphone back on. so i'll talk a bit aboutwhat we need to look at in terms of network assessments here. so as you might think,networks are quite important for cloud services to work well.
so we always encouragean assessment before any move to the cloud, any change inyour connectivity to the cloud, and that would include puttingan express route circuit in. so you've got a full end to endview of what your options are, what the bestconnectivity methods are. so, from a microsoft perspective,we offer a number of services too, to do this. we have a massive amounts of partnerdeliveries that would do this for you as well.
to ensure that your connectivityend to end is optimal as possible. but which kind of service you needdepends on what you are doing. bandwidth planning is alwaysthe first thing that should be done because, one, it takes a long timeto do, and two, it takes a long time to order new bandwidth forcircuits in a lot of cases. so from a microsoft perspectiveswhere americas cloud services, mcs might be able to come in anddo this work for you. we published calculators online tohelp you understand what bandwidth is used by services.
but it's, those are only as goodas the data you put into them and we generally recommend usingmeasurement of your pilot users or measurement ofthe traffic going in and out on your on-premises exchangeenvironment and mapping that out. and using that data toput into the calculator. in terms of connectivity, makingsure that your sites are connecting to microsoft in the best way andthat your egress equipment isn't changing things that are causingproblems or causing bottle necks. premier due deliver a networkperformance assessment where
they'll look at a numberof sites for you. and say this site is notconnecting optimally. it would be betterto switch it here. or this proxy is doing this to yourtraffic which is throttling your through put etc. so that would be, if you are havingperformance issues, they would be able to point out where that isor it's worth while running those in pilots to make sure that yourconnectivity model is optimal. when you're looking at skype for
business, that needs a veryspecific assessment. because the skype havean excellently detailed framework. the skype for business framework, i think there was a sessionearly on this week on it. essentially, they will map what therequirements are in terms of network connectivity fora high quality call video meeting. so that means from the clientto the edge of your network, x amount of latency,x amount of jitter, etc. from the edge of your network tomicrosoft's network should be this.
and then end to end, and they've published a toolearlier this week to do that. publicly available tool where you, the tool will check allthose things for you. so it's worth running that yourselfif you're comfortable with that. or premier should have a deliveryvery soon, where they'll come in and do that for you andhelp you understand the output. so those kind of assessmentsfit different requirements, but when you're looking at expressroute, you really need to be
thinking about how each of yourphysical sites are gonna connect, once those circuits are put in. where does circuits need to go,where's the optimal place to put them, and whether you've gotoutline sites, that it does not make sense to use expressroutekeep them over the internet. this sort of planning needsa network assessment to figure out where those lines lie. so i just talked about bandwidth interms of how we recommend it, but there's a great link therethe aka.ms/tunemsit,
it goes through howmicrosoft it planned this. as you can imagine, we're quitea heavy user of our cloud services. so smit came up with the approach, it's outlined indetail in that link. but essentially theycame up with a figure of. >> 400 kilobytes. >> 400 kilobytes a second. and then they just extrapolatethat out by the amount of users in that site that they're gonnaconnect, and they work off that.
that works out to a veryhigh level of bandwidth, which is what we work to. and if you have two users in youroffice, it doesn't work overly well. but it's a nice starting point ifyou're an enterprise getting toward the size of microsoft and usingthe level of cloud services we do. and the skype for businessframework, i mentioned before, is also linked there. again, if you're looking at skypefor business online, it's absolutely necessary to work through thatto make sure that your call and
audio quality is gonna be asthe skype team intend it to be. so security, i mentioned thisearlier on, it's a common discussion i have to have with customersaround when they want expressroute, whether this is a driver forwanting expressroute. they don't want theirtraffic on the internet. so we address this because,the security required for expressroute, it does notgo away as a requirement. so if you look at that topbulletpoint there, the way we see it is the internet requires yourhighest level of security policy for
obvious reasons. that's what you already do. and you're doing it very well. office 365 expressroute sitssomewhere in the middle between that and connecting to youron-premises datacenter. sort of a one link outto your datacenter, or azure is, where you'd needa limited amount of security policy, cuz of the way that's connectingto an isolated environment. so office 365 sitsin the middle there.
so it's not somethingyou can just forget, but it maybe doesn't need the levelof intrusion detection, all that stuff that is doneat the internet level. >> and i think we kind of see this,because most organizations are familiar withinternet security requirements. and, ots of companies have a policyaround network security for connecting onto the internet. and there's typically a lot ofdetail around that policy of what the requirements are,what kind of devices are required.
and it's often you spenda lot of time on that. and what we're saying is,well you're not connecting to the internet in this case, you'reconnecting to microsoft data centers and don't you trust microsoft? so it sort of stands to reason thatyou would have a different security policy for doing that thanconnecting directly to some random internet site which, of course, isa policy people are familiar with. and i think what we tend to seeis because it's different and it is a lower security policy,
we see people go to, i don't needany security if i'm doing that. and what we wanna tell you is thereis a level of security that you wanna think about. you wanna understand what are allthe risks that you're protecting against on the internet. some of those, maybe,have gone away. microsoft is a known company. you have contracts with us ifyou're connecting to our servers. and sosome of those risks have gone away.
other risks, which you havea security policy, and probably have not. so, this is different and maybe adifferent level of security policy, as paul was just talking about,is needed and something that is a little bit oftime worth thinking through or you can ask us fora device about what we think. but it's not just no security and it's not probably the samesecurities you have on the internet. and often i run into things where
customers are trying toimplement security policies when it's worth having a look at whatmicrosoft does in the back end. are we providing something on theback end in terms of security that negates the need for equipment atthe edge of your network to do that? cuz that could cause a bottleneck. it's always worth looking at this indepth, and what we're doing and what you're doing and making sure thosetwo things fit nicely together. so expressroute for office 365,it doesn't prevent ddos attacks. what we're referring to here is thatthe internet is still required.
so let's say you'reloading a sharepoint page. elements of that page are notyour data but scripts. generic images that are not yoursmight be loaded from a cdn over the internet. so if that internetpipe is blocked for whatever reason, then you'regoing to get performance issues. even though you canget your data for the page overthe expressroute circuit so. doesn't prevent thatfrom being a risk factor.
we don't lock down, well there isno way to lock down given it's a public distributed architectureaccess to other office 365 tenants. so you'll use this in theory fromtheir machines to connect to any office 365 tenants. because it's an end to endconnection to our network and all our services. it doesn't replace the need forproxy servers if you use them. load balance is firewalls thoseare still required on the internet pipe and to a certain degreeon the expressroute circuit.
the circuit also doesn'tprovide source nat. so your connectionscoming in from microsoft needs source maxing in most cases. and your connection going out fromyour environmental need nothing on the way out, because you got toaddresses with public ip space. and if you have specificinspection filters running for whatever reason it doesn't replacethat but sometimes it might do. that dlp in our back end markprevent you're having to do that on your environment asi said earlier on.
and what are those pitfalls we seecustomers run into quite a lot is using the url and ip list pagethat we have to block access to anything that's not on that pageto the expressroute circuit. that should not be the waythat you manage this circuit, because that page isnot dynamic enough. we try and update it well inadvance of any ip change. but given the way bgp works and occasionally we have to do emergencyip changes for whatever reason. then that's not a dynamicenough process, or
your security team might be onholiday and miss the oss update. these things happen, andthey do happen a lot. so to protect that circuit, you'vegot multiple ways of doing that. you can use max routes, if you'reworried about the route floating. you can limit that tomicrosoft's asn or we publish a full list of our crgrs. so that's every ip address microsoftowns, so you can use that as a list. that doesn't change in any waynear the rate that the url and ip list for office 365 does.
>> yeah you can't really getmore ipv4 fortress this days. no, not cheaply. >> so, and that policy needs to be down on every expressrouteyou're looking to implement. so you can see thisstuff takes time, it's not as simple as turningthat circuit up as i said. so the way we just described today, you got three types of peeringwith microsoft expressroute. you've got azure private pairingat the bottom there, which is
essentially an end to end connectionto your constrained environment. that is where you would needthe least amount of security, because that's an end to end tunnelto your restrictive environment. in most cases, you're not able toget out to the internet from there, unless you've got a route out. the azure public pairing ormicrosoft pairing, as i have just gone through. need a level of security to ensurethat your environment is protected when you are connectingto those two services.
okay, all right, that's security. we are going to talk aboutan implementation plan. so the idea with an implementationplan is we just want to think about writing down what we're gonna do for this routing changes beforewe go here and do it. so we got a plan forbandwidth requirements, plan for what we're gonna do about security. what we're gonna about the highavailability requirements and failover.
and then we wanna havea design written down for outbound connectivity. and then for each of the inboundservices that you have for the organization. all right, so,we're gonna write that down so that we can think about it. maybe have someone review itbefore we decide to go ahead. how far is expressroute routeadvertisement going to be propagated within your network?
this question is really about areyou using bgp across your network? or are you just receiving bgproutes at the network edge and then using some other kind of protocol orsome other mechanism for having that route and get back to the pcsthat are in the organization? so i'll talk about the most commonways we see this being done your options here in the nexttalk in more detail. so then dns,you're gonna have some changes for your dns related to the routingchanges potentially. your net strategy,paul just talked about net.
you've got to have net in orderto provide public ip addresses. there's a few restrictionsexpressroute has around the ip address is a net polling andnot sharing those. and that's a gain related to so thatwe can have route advertisements for those ip address is goover separate circuits. and not have a request and responsesgo over different circuits. the network topology update. it's always good todo a paper-verify of these network topologies.
sometimes we'll even do this onthe phone with a customer if they're having a problem. ask, okay, where does your inboundservice for adfs hurt your network? what is the server? so this ip address,it's in this location. what kind of routes are exposeto that server within your organization? where does the traffic go afterhits that load bouncing server when it comes inside you network?
paper verify is justessentially by about thinking about each of those inboundand outbound services and saying. okay what hops does thatmake across my network. and making sure that the routes thatare available to each of those nodes are what you expect. and when you do a traceroute that's going to match what you have donewith paper verify. i like paper verify because you canactually follow it all the way. whereas with trace route, very oftenyou get blocked here and there and
you just can't follow it. also you're only followingthe forward path. you're not seeing the reverse so, although it's real it is verylimiting in what you can see. all right, soavailability and performance. dedicated circuit does notequal a security barrier and does not equal high availability. so we talked about thisa little bit already. you want to plan forthese kind of things and
make sure that your plan hasno single points of failure, assuming high availabilityis critical to you. and make sure that if there isan outage that you're minimizing the impact and minimizing the blastradius of the outage that you have. all right, let's move on,we're talk about perimeter networks. >> so there are some of the designthings you need to think around perimeter networks. we've talked all ready about wherethose meet me locations are. trombone route avoidance,that's referring to,
what you don't want is put a massivelongate pipe in somewhere, and forget about the local land links. because that's gonna havea bottleneck that goes through and then spits out into a large circuit. that circuit can be large enoughto cope with all the traffic, but if the end to end bandwidthavailability isn't good, then we're gonna haveperformance issues. using multiple meet-me locations,absolutely recommended. can you use a singleexpressroute circuit?
yes. should you? probably not is the answer. or at least have a very good backupplan to switch over to the internet. so have a look where thosetermination points are. obviously there's no real pointin putting two expressroute circuits into the samemeet-me point. so some cities,london being an example, have multiple places whereyou can run circuits into.
or working with other customersthat have one in the usa. and if the europe one goes offline, gdp routes will switchthat over to the us. it's not an idea of scenario, but hopefully that down time in thateurope circuit is pretty short. but we're not gonna have a completeoutage of connectivity during that period, we can still connect tothe flow that were doing already. so these things needplanning think about, this were your network assessmentwill show you were that
would be necessary. and of course the internetis always an option for you. if you've got that capabilityto switch that over correctly. and avoid routing trafficto different continents. i'll give you a story. i worked with a customer the otherweek that put an expressroute circuit in the usaon the east coast. and they turned that up, andthen they set the pgb routing up to their apac site to usethat expressroute circuit.
previously usingthe internet in apac. so we picked that traffic upin apac on our network and get it to where it's gotta go. now when they put that circuit live, all that data in apac was having totraverse their mpls over to the east coast of the usa to go out to theinternet to hit microsoft's network. and then go to where it's got to go. so that is not optimal connectivity. and they had some seriousperformance issues there.
partly down to their dns setup, which we'll kind of cover laterin this one or in the other one. i can't remember wherethat slide is, but it's around how outlook triesto connect to local endpoints. but you can see the point here, that we've got to thinkabout this traffic flow. and insure that we're not actually,putting this premium circuit in, we not actually causing a negativeimpact to some of our users. we've already talkedabout inbound services.
you've gotta understandevery inbound services. as i've said we'll talk aboutexchange in detail later on, around what needs to beexposed to the incident. because for example, if you don't expose autodiscover to the internet, your users aren't gonna be ableto connect and get their mail. so their are considerationsaround that. so we need to understand the trafficflows and what happens if we don't expose those to the internet andwe use express route.
outbound services,we thoroughly recommend using separate mac calls on allyour expressroute circuits. so no overlapping macon the internet, and your expressroute circuits. why? it protects you from asymmetricrouting, it means traffic that goes out of that circuit will besent back to that circuit. and it ensures that your firewallsaren't gonna run into that scenario where they don't see the outboundpackets, so drop the inbound.
and internet andegress requirements. i'll talk about thisa bit more in detail but you've still gotta take careof that internet piece. it can't be forgotten whenyou're moving to expressroute, it's still vitally importantto your performance. there's my slide iwas thinking about. geo dns, so i explained to you inmy session yesterday in more detail around how outlook, we havea geo dns system which allows outlook to connect to a cas serverin the region where the user is.
and we do that by lookingat the dns call and figuring out where thatdns call was sent from. and we can say,okay this was done in europe. so even though your tenantmight be a north american one, that user in europe, your office in europe will hit a casserver in the european data centers. and then we're back all the datafrom there now, that works nicely if that dns call is beingcompleted in the right place. so imagine that scenario ijust talked to you about,
where we split up an expressroutecircuit in the usa. so that's my incident egresspoint for my apac users now. but, what they have been doingit's changed the dns call so it completes in the usa. so what was happening isthe dns call completed in apec we get a list of servers in apec, but now i've changed my egresspoint to east coast of the usa. so the traffic has to go tothe east cost of the usa, into microsoft's network where we look atthe end point, say that's a apec so
we connect it all the way back toapec where that user first started. that is massively sub-optimalactivity as you can imagine. so these things needto be thought about. i have a blog post which is linkedon the tune site which should be linked at the end, which walksyou through how to check geo dns. the simplest way tothink about this. your dns cost for outlook to office365.com needs to be resolved in the region where your egress is. and that egress should be in theregion where you uses are anyway.
i kinda touched on this earlier on. so snat is thoroughly recommendedfor incoming connections. so we can ensure as i say its a surefire way of ensuring that traffic gets sent back the same way becausewe have to respond to where that address is, sofrom an exchange hybrid server for example, something comes infrom office three six five, that source net device will flickthe ip address to the internal address of that source net deviceand send it to exchange hybrid who will then respond to the internaladdition of that snat device,
which will then send backout the same circuit. if you don't do that, imagine a scenario where itcomes in over the internet. we've not source natedthat connection. so when exchange sends the responseback, we'll see an optimal route via the bgp circuit, theexpressroute circuit, to get back. and that's wherethe traffic will go, and you'll run into thisasymmetric routing problem. i mentioned earlier onabout the url and ip list.
so this might be your pac file,or any devices using that on the internet pipe, for example,need to be kept fully updated. cuz if you miss things off, then you're gonna getintermittent connectivity issues. to help in that area, i've writtena pac file, which very clearly shows you which urls are required forexpressroute and the internet. because it can be a challenge,as a lot of you are aware. who's looked at that url andip list? well, some of you.
well, there's a big list there andit's very complicated cuz we have to expand those out per service andthere's a lot of overlap. so anyway, i've written a pac filewhich very clearly outlines what's required forexpressroute and what's not. or what can be sentby expressroute and we'll publish those either the endof this week or early next week. you should be able to just grabthose, and we'll keep them updated, when that page gets updated. >> i'll just talk aboutthe smtp inbound mail briefly.
if you run an on-premisesmx resolved smtp server, then you should not advertisea route to that specific server, to microsoft over expressroute. you need to advertise thatover the internet and the reason is, although ourexchange online servers, you will get routesadvertised over expressroute. you can also connect to themover the internet, of course. you will not get routes advertisedfrom any front-end servers for microsoft hotmail orconsumer email services.
and so if you advertise your smtpserver over expressroute to us, that route will be part ofmicrosoft's core network. and so those consumer emailservices that we have, which also run on that network. we'll be able to see that route andthey will try to send any email from microsoft consumer services toyour smtp host over expressroute. and so what's gonna happenis the response for that send packet, the response isgonna go over the internet because you do not have a route to ourconsumer services over expressroute.
and so that's going to resultin email not getting through. troubleshooting a mail notgetting through is very different to troubleshootinga route error. and so we have to make sure that youadvertise those over the internet. so that's it. a good tip forhow routing can get much more complicated than youexpect pretty quickly. >> and we'll talk a bit more about thatexchange flow in the next session.
so i also touched on, earlier on,around restricting that traffic coming in from expressroute,securing it. so, as i mentioned, don't usethat url and ip list to do that. because it's not dynamic enough. there are various options i'vetalked about before using the full range of ip addresses, et cetera. what you can do ituse bgp communities that are listed as preview,but they're working in life. and you can use those to restrictpackets that are tagged with
microsoft's office 365 communities. so i'll talk a little bitabout your lan choices, cuz this is gonna be coveredin the next session. but what do you still needto get to the internet? dns name resolution,for obvious reasons, that has to get out to the internet. crl checks, again,for obvious reasons. you'll see most of these are notheld in our data centers because it doesn't make sense to.
so office proplus, cdn data, crl checks, we can host thatmuch closer to you in cdns. we use our own atomize as well, andobviously there's no point routing that into our network and out,it has to go out to the internet. yammer, because of the wayit's hosted currently, has to go over the internet. office video, because i thinkthe data's held in cdns, that's the reasoning forthat, or some of it. >> it's in azure public.
>> that was it. >> storage. >> so these things obviously needto be mapped to the internet, and you need to, if you're using a pacfile to separate that traffic, we can take care ofthat in that platform. as i say we just published them orabout to publish them. so your connectivity options,how do you get your traffic? you put an expressroute circuit in. how do you get your trafficfrom your client to that
expressroute circuit? and this is the complicated piece. i sat down, we workedthrough what the options are. there's thousands of different kindof methods that you could use and every customer'snetwork is different. but if you boil them down you canhave direct network connections with bdp, direct network connectionswith default routes. so we make a routingdecision at the edge. or we use a pac file to send thatexpressroute traffic to a proxy.
so, i'll run through these inmore detail in the next session. >> all right.validation, test and implementation plans. >> i was just gonna say, if youcan't make the next session, and there's another one you want togo to, these are all recorded and available online. so, you'll be able to watch it atyour leisure, at a later date. so the thing about making a routingupdate is, it's like anything, you've got to make the update andthen you've got to figure out well,
did it work? and sometimes the issuewith make a routing update is you're actually doing iton your production system, so i wanna talk a little bitabout testing for a moment. and part of your implementationplan needs to be how am i testing that my routing changes that i'mabout to make are going to work? are going to result in myuser still being able to connect to the internet, still beingable to connect to office 365, that all my inboundservices are going to work.
so what are going to do? i've got four ways thatyou can test those routes. the first, and i mentioned thisearlier, is to do a desk check. so sit there with yournetwork topology, with your routing information,couple of people, get them together. talk through what your routesare for each of your outbound and anything that's different,talk through them one at a time. make sure you can convince someoneelse that they're correct. the second way that you can dotesting is to test on a test
network. this i do find, we don't findtest networks very often. some customers willhave test networks. we're basically talking about, you have a separate connection tothe internet from a test lan and a test perimeter network anda test expressroute circuit. a lot of people don't wantto invest in doing those. we do come across them now and then. i've seen a few recently wherethey use various methods.
they can put that circuit,the expressroute circuit live but prevent any of those bgp routesfrom filtering into anywhere but a particular segment of theirlan which is used for test. if you don't have one they'repretty expensive to set up, cuz you're duplicatinga lot of equipment, but a test network means you can makeany routing changes you want and you're not impactingproduction users. the third way that you can test is,if these are just routes that are gonna impact office 365 andmicrosoft data centers, do it before
you deploy office 365, because thenif you make routing changes and they potentially impact an exchangeserver or something and no one's using exchange yet,it just doesn't matter. you can do this kind ofpre-production testing as well. the fourth way, which is a kindof the catch all, because you can always do this kind of testing,is to test during a network outage. some of the best deployments i'veseen, have been where people have planned for multiple networkoutages on weekends. like four weekends in a row,they will plan for
a multi-hour saturday morningnetwork outages, where no one can be in the office using the lanbecause we're changing the network. and the first time you go through,you make your network routing changes then you go andsee what worked, and you find there's a wholebunch of services don't work. so you write those down,you put the network back and people come in monday morning. then you've got another week togo through and figure out, okay, what did we do wrong,what do we wanna change?
and then your next weekend comes up. if you plan for those in advance, then it's a greatexperience when you go through and you did what you said you weregonna do, you had a network outage. you found some things that werewrong you put it all back and your users are okay again. and you can hit your deadlines forgetting your network up that way. it does take a little bit of time,because you've got to plan for all of these things, but you canalways do that regardless of whether
you have a test network orany of this other kind of stuff. so four types of testing,you absolutely want to do testing. if you don't do testing, there's areally good chance that you'll make these routing changes, andit won't work as you expect it, and you'll start to get thosesupport calls come in, so i can't stress this enough,testing before you go live. the last thing on implementationis just to have a plan for deployment and validation. after you do these routing changes,just go through and
test all of the connectivitythat you expected to have. this is where it's also valuable tohave a list of inbound services. you don't actually need totest a lot of different users can see an email. you just need to test thatone user can see an email. and then you wanna test thatthat user can receive email. so test all of thoseinbound services. you don't need to have a hugebroad test there as well. and you could use the pack file thatwe're gonna publish which lists
those urls very clearly withthe expressroutes with urls, and just do trace routes to themfrom your different network segments to ensure they're traversingthat bgp circuit as you expect. >> all right, andour last step is troubleshooting. and i'll hand over to you again. >> so some of the commonthings we run into where we have to help customerswho have issues here. we've mentioned it multiple times,we're not mapping out or not knowing about thoseinbound services.
so, that should be one ofthe first things that you do, make sure you fully understandwhat's open over the internet, are you going to keepthem over the internet, do you want to move some tothe expressroute circuit? if you do so,what is the impact of that to your users thatare outside your environment? a common one we saw working as well,another driver for expressroute use is they wantto stop using the internet, that is still requiredas i've mentioned.
the vast majority of our coreservices, your mail flow, your sharepoint data,your onedrive for business data, your skype flow will go overthat expressroute circuit. and we explicitly outlined on thaturl and ip page what does and does not go over expressroute. kind of talked aboutit a bit earlier on. for the majority of servicesthat require the internet are things that aren'thosted on our data centers. an open routing network, sowe'll talk in the next session about
the complications ofallowing bgp routes all the way into your environment andthe extra work that needs to be done to ensure that the traffic goesout the way that you want it to. not using source nat, again, this is a real common causeof asymmetric routing. i cannot count how many timeswe've run into this problem and support has had this issue. and it's very hard with someone thatdoesn't have a technical network background to actuallytroubleshoot this.
they just know it doesn't connect. and they follow the path they expectand eventually we figure out, okay, it's coming in here andgoing out here. so using the correctsource nat pool and our online guidance walks youthrough what's required here. if you do that correctly, you should prevent thatissue from ever occurring. not testing, as paul just said,common where we fall into things that we would have picked up duringtesting, if that had been done.
and a rare thing, but. >> maybe only once. >> paul's had it once,i'll let him explain. >> so please don't use an azuretrial subscription for your expressroute circuit. this may have only happenedto one customer ever, but i did have to help them with it,so i'll tell you the story. if you use an azure trialsubscription, these things expire. and so i was working with thiscustomer, and they called up, and
nothing was working. their circuit had gone down. it turns out what happenswhen the azure trial expires is your expressroutecircuit will get torn down. and because this is allsoftware-defined networking, the network provider circuitwill get torn down as well, everything goes away. you have to start again and createall of these things from scratch. so please try not touse an azure trial.
at least don't have an azure trialsubscription that expires on the day you're trying to go to production. that was the issue here. >> [laugh]>> i'm sure that will not happen to any of you. >> paul gets all the fun ones. >> so, a baseline of connectivity,so regardless of route, connectivity to office 365services should work, so we, as i said before, we've gota public ip infrastructure and
the expressroute circuit ispurely a routing override. sometimes when i look at this,if we miss a ul in a pack file, what happens? it should, hopefully, be caught bea catchall and sent by the proxy. so it'll still work until somebodycatches that and fixes it. that's the kind of scenario we wantto be in, and when you're designing a network flow,if something is missed, hopefully that's gonna switch to your internetpath and then it should still work. the url and ip list i'vetalked about 1,000 times
is the link there, and it has acolumn in there that says yes or no, whether that service, that url,goes to expressroute or not. we can route it via expressroute. and as i said, we're going topublish that pack file which, cuz that page is enormous. and you should be able to justread through that pack file and see the consolidated lists of urlswhich can go by expressroute. >> okay and in terms ofthe actual circuit itself, often when you have a problem getting thecircuit up and live and provisioned,
it would be a support call to oneof our azure networking team. and they should hopefully relativelyquickly be able to fix that for you. the only state you want to look for in terms of that connection beinglive is provisioned and enabled. how do you know it's working? a trace route, see if the traffic'straversing those particular routes. you should see the routes come in,the bgp routes, come in to your environment. there should be around 600 come in.
your network team will certainlynotice that when it happens. psping to office 365 resources. i'm not sure that's a really goodtroubleshooting step because it could be that your traffic isn'tgoing via expressroute as you expect, but it is connectingover the direct internet path. so but at least it shows that thatservice is connectible in one way. >> it would be badif it doesn't work. >> yes, exactly. a traceroute would beyour better option there.
prefix validation,i'll talk about here. we obviously check thatthe addresses you are advertising to are owned by you. and if that's not done,you can't make the circuit live. that is pretty instantaneous. we check with a multitude ofplaces to ensure that's the case. but if you're in a situationwhere you don't actually own your ip address space, you lease themoff an isp or someone else and they're registered with someoneelse, it's very wise to do a support
call to microsoft in advanceof putting the circuit live so we can do those additional checksfor you and confirm with the owner that it's okay forus to let you use that address. and that's for obvious reasons,that we don't want to allow somebody to use somebodyelse's ip address on these circuits. general connectivity failures, quite often these are purelyasymmetric routing. again, if you do that source natcorrectly, that shouldn't occur. over deliver those, as we say if thetraffic isn't going via expressroute
circuit, for whatever reason you'rerouting or your pack file is incorrect, hopefully thatwill go via the internet. and you can pick that up andthen switch that over and resolve the issue. and with that, here are some ofthe links to walk through this. there's the implementation guide. we've got the second link there,it's very detailed. and the whole team of us wrotethat based on customer issues, successful customers.
and it's a really great resource foryou to just step through and it might make you think we forgotthat bit and walk through it. so i thoroughly recommend havinga good read through that and it gives examples of natexamples through expressroute and all the other complications andissues and the things we just talkedabout in the last 70 minutes. with that, we have about fiveminutes left of questions, we should be here forabout five minutes afterwards but then we've gotta hop overto the other building.
but, please if you have anyfeedback about the session, please fill in the eval and ifthere's any questions please step up to the mic and we'll answer them,otherwise we'll hang around here for five or ten minutes andanswer them one to one. >> you can also contact us later. on twitter i'm @pndrw,love to hear from you guys. love cloud networking, reallyappreciate you coming along and listening to us talk about it today. >> [applause]
