15 year CCIE!

I just got an email from Cisco congratulating me. 15 years wow how did that happen?

I think back to where this all started. The HP 85. After college, I spent years working in electrical engineering for many companies on various DoD products. The HP 85 was very popular in the early 80s as you could write programs using “Basic” and pull instrumentation off the serial bus of any HP instrument. I did a lot of R&D and also QA so it was important that we documented the results of all the testing.

The successors had more modern color screens and capabilities and this was a large part of my work environment.

I was fortunate to work on avionics for the F15 and A10, precision cesium standard clocks for the Space Shuttle, Project Gallelo, and other satellites. Eventually, I made my way to Sperry Gyroscope (Unisys) working on the MK-92 fire control radar. We still used the HP85, but then DOS on the IBM 8086 came around and we had serial cards that could integrate with the testing equipment.

Eventually, my future electronic jobs started to use windows 3.11 and then I can remember converting dozens of computers to windows 95 with 15 floppies. Took forever. Back in the mid-’90s the “certification craze” took effect. The big one was the MSCE, and my employer Brookhaven National lab was offering it at night for free. So I took the courses and learned all about Windows domains, IIS, and Exchange and got my MCSE in NT 4.0. I even built my first little network using a 486 as a router.

We also had some Cisco routers to connect building out at the lab and I was interested in that as well so I studied Cisco and networking and got my CCNA back in 1999. From there it was hook line and sinker. I went down the Cisco and Microsoft path learning as much as I could using my home lab. Eventually, I was able to get my first real IT job(even though I have been doing computers and such since the 80s).

At my job at Northfork Bank, I was responsible for all the networking as well as computers. In those days it was token ring for the workstations and DLSW over fractional T1 or ISDN to the branches. By then I had my CCNA/CCDA and CCNP/CCDP. The next step was the CCIE! This was 2001 I attended Bruce Caswell CCIE BootCamp, read all the books, and took the exam in early 2002 and failed. Also in late 2002, I got a job with LendingTree down in Charlotte NC so I moved down and my family came down in 2003. At that point, I had a good job and the CCIE took a back seat while my kids were young and growing up.

In 2006-2007 I started collecting gear so I could work on the CCIE R/S test again. I passed the written and used CCBootCamp and IPExpert materials. I had my lab on the dining room table for the longest time and one of my neighbors came home with a sweet rack with glass doors someone was throwing out. At that point, the CCIE lab reservations were actually running on the weekends. So off I went to RTP on Saturday, Feb 9th, 2008, and spent the day in a hotel room cramming for any last things.

Sunday a group of CCIE candidates was waiting to get in. Most were from all over the country and there was a guy from the UAE. I breezed through the first part and everything went well up to lunch. One guy walked out at lunch never to return. I can’t reveal details but there was an easy 5-minute configuration I could not get working. I spent a great deal of time as I couldn’t move on to further items. I went to the proctor and asked if was it possible somehow I was hooked to the wrong link. Nope, all things looked good but he did say you are missing something really basic. I went back and instantly saw the deception and my neighbors came up. I had about an hour left to do 3 hours of test so flew through it and finished.

The proctor asked if the 10 of us left wanted to wait and if he would grade it right there. Of course, we all said yes. And we did it alphabetically so of course, Witte is last. One by one people were graded, failed, and left. Then it was my turn. Before he started grading he mentioned how well I finished off the last portion after getting stuck he never expected me to pass. And we went through my test and I passed easily. He handed me a yellow tear-off pad with my CCIE #19992! I even wrote my name up on the board there in RTP. Out of 10 or 11 of us, I was the only one to pass. Literally one of the proudest moments in my career. Still have that piece of paper.

The proctor then said since you did so well on some of the sections, I would go home and start studying Service Provider you should be able to get past that easily. So that’s what I did buy more gear, an ATM 1010 Lightstream, some 7200s with ATM, and some IPExpert SP material. I tried 2x for the SP CCIE and that was when there was rampant cheating. There was a 4-question written portion worth 20 pts if you got 2 wrong you started the test with an 80. And you need an 80 to pass so both times I got a 74 so I should have passed. I did that 2x and didn’t get through the 4 questions pretty unfair for an at the time $1400 test to ride on. Then they changed the test from IOS to IOS-XR, new gear the whole thing was changed. At that point, I had enough with SP. Below is my lab with the CCIE DC added. NetApp filer, MDS switch, and servers for virtualization. You can also see the LightStream 1010 on the bottom from the CCIE SP.

Here is the entire lab, I had frame relay, ATM, and call manager, then used VMWare a vASA, virtual 7k running OTV, FC with the MDS, and a UCS simulator. I tried the CCIE DC using this gear but failed as well. At that point, I had joined WWT and was traveling a lot so with the travel and family obligations I put it away

Here is the physical portion;

And the virtual portions with the CCIE DC gear and virtual 7K and UCS

I’ve tried seeing about revisiting these but 15 years have gone by. I’d rather learn AWS and public cloud, Kubernetes, smart NICs, and AI/ML. The CCIE was a great gateway to getting me fantastic jobs at ePlus and WWT. People ask why I don’t go Emeritus and that I’ve worked too hard and you never know where you’ll end up. I keep doing my re-certifications using CE learning credits instead of the CCIE written. I’m up to 50 credits, I need 70 more CE credits from Cisco to re-certify everything.

Been a long 40 years journey with computers then moving to networking and I’ve been fortunate to see it go from AppleTalk, IPX/SPX, RIP, DLSW, Token Ring to SD-WAN/LAN/DC, AI/ML training of the network and virtualized everything with VNFs. Now we are getting into containers and microservices via Kubernetes and other container platforms so there is never a lack of something new. Next is 20 years!

Ask the customer, Cisco Nexus Dashboard

Please join Cisco and WWT for a webinar showing customers experience with Cisco Nexus Dashboard and Insights, real-life use case’s and how WWT can help you get Cisco Nexus Dashboard and Insights into your infrastructure to monitor your network for a free trial run proactively. Learn new features from Cisco and how WWT can help get customers onboarded for the trial. WWT also offers an SKU-based service that many customers have leveraged to help set up Cisco Nexus Dashboard and Insight and create use cases to evaluate the product in your network.

If you feel up to doing it yourself, navigate to WWT.com and look for the two-part white paper for installing Cisco Nexus Dashboard and Insights and creating 20+ use cases. The first part of the white paper shows installing the physical or virtual form factors nodes to build the Cisco Nexus Dashboard Cluster. Then we onboard sites and any 3rd party Apps.

Part 2 of the white paper contains over 20 use cases, so a customer has an easy way to validate and test the ND and Insights in a trial period. Most customers that we have helped install the Dashboard or used our 2 part white paper found Cisco Nexus Dashboard and Insights so valuable that they purchased the licensing and are delighted with the accessible troubleshooting insights.

In the webinar, we see a real-life WWT customer scenario of changing their troubleshooting time from hours and multiple teams to 10 minutes and a single team (Netops) using Insights. We perform a live demo showing how that is accomplished in these day 2 tools, reducing MTTI (Mean Time To Innocence) and MTTR dramatically.

Also, for our customers that are replacing vPC-based 7k/5k/2k 3-tier networks with Nexus 9300 and 9500 switches, we can now leverage the functionality of Insights to these devices even though they are not ACI or in an EVPN fabric. Most customers assumed that Cisco Nexus Dashboard and Insights were only for ACI fabrics. What most Nexus 9k NX-OS customers dont know, is that you can gather most of the telemetry of what ACI can send to Insights with NX-OS-based switches. You need Nexus Dashboard Fabric Controller (NDFC) to proxy the telemetry to Dashboard as a site and eventually into the correlated database that can be used by Insights the same way we do with ACI. Integrating your NX-OS fabric is very simple, and you can gain unprecedented visibility from your existing NX-OS-based 9k-based infrastructure. Instead of needing a team for initial troubleshooting with Insights, you can quickly find the root cause, rule out the network, or find the issue in the network.

You can sign up at the link below and join Cisco and WWT for a deep look into how Cisco Nexus Dashboard and Insights can provide faster MTTI (Mean Time To Innocence) and then find the root cause, make the changes and restore the outage in 10 minutes instead of all day and engaging multiple teams.

Please sign up for the Webinar on Sept 21st from 10:00-11:00 PDT and hope to see you there!

https://webinars.cisco.com/amer/insider-series-cloud-nexus-dashboard#xd_co_f=MDEyNjRiZTAtNjA4Yy00MzY4LTkxOGUtMDIzY2Q0ZmQyNjM0%7E

2014 and moving to the national team at WWT

2014 i became more involved with the national team as well as managing some of the data center redesigns we had in flight. Since my role is a pre=sales architect, we would get it to the HLD and BoM phases in a design, then professional services would take over the day to day implementation and migration phases. Of course since its our design, we need to remain plugged in to implementations in flight as well as new opportunities. As I am traveling around the southeast helping account managers close new business I am also supporting existing implementations in flight. Now thing really start to get hectic traveling and meeting customers by day and managing projects in hotel at night. Lots of travel across south as well as St louis and the west coast.

I was working on my VCP5-DCV cert plus i started working with the national team as Cisco had just released the alpha code for ACI(before the first ACI code). We had some amazing ties into the Cisco Insemi BU at the time, and had gotten the first 2 NEXUS 9508’s off the assembly line back in Nov 2013. With this new hardware we were able to install the first ACI code, and worked with the BU as they started releasing the beta code to partners. Once ACI was introduced, the national team created a ACI class to help customers understand ACI. I worked with them and got trained so I too could give the class as a overlay for the southeast. I was also working with the Cisco folks on a UCS director now that Cloupia had been purchased by Cisco. I was familiar with Cloupia and had worked with it before but now we were pushing it and had to be a expert so went to some bootcamps and training.

In the Oct/Nov 2014 time frame i was approached by the national team to join them and focus exclusively on NEXUS, ACI and data center switching technologies for the entire country. Now its really going to get crazy.

Home Lab build

So I have been labing stuff up forever. I did it for a living for 20 years working on new electronic designs ranging from a crystal oscillator that’s on the Galileo probe that’s past our solar system, new fire control radars for our military, RF acceleration and control systems for particle accelerators looking for the Higgs Boson, even upgrading car stereos for better bass. When I wanted to learn NT and computers I built out a domain with webservers and exchange running on some 486 boxes and token ring. My CCIE adventures got me buying ISDN simulators, old CAT OS boxes, ATM and token ring to build my CCIE labs. Now with virtualization I have been able to create a killer lab to really have some fun. I have always been one that needs to take something apart to see what makes it tick. I honestly can say that this helps me with my role as a pre-sales engineer. I feel that having hands on experience with technology that you are trying to sell is very important to closing the deal for you company. I have won many deals by working with the engineers at my customers and drilling in and showing what really under the covers, how it breaks and how to fix caveats etc.. You had also better be able to balance that with talking the business aspect with the CIO or architects who really don’t want to get in the weeds. A very fine line presales has to run on.

So my lab physically consists of about a dozen routers ranging from my lowly 2511(a great reverse console box also good for injecting BGP routes and whatnot to 3600’s, 2800’s and 7206’s(for my CCIE SP studies) I also have a nice Netapp filer, a MDS switch and 3 Esx 5 hosts booting from FC off the Netapp SAN. My virtual environment I built to support my studies for the CCIE datacenter and I have CIAC, Cloupia, the 1000v, the ASA 1000v cloud firewall, VSG, VNMC, a couple of UCS simulators managed by UCS central, a EMC VNX simulator. I have a couple of NEXUS 7k simulators running OTV and some vWAAS as well running in there. Finally I just got finished getting the beta version of the 1000v on Hyper-V running I had to stand up a couple of Windows 2012 Hyper-V hosts with SCVMM on one, but had some issue running nested VM’s due to the lack of my processors support EPT which is needed to nest 64 bit machines on hyper-v(or anything else really) The workaround was to run the Hyper-V VSM’s on the ESX hosts, and then install the VEM’s onto the Hyper-V hosts running as VM’s on the ESX hosts! Pretty crazy but it works. Not very fast but I am able to through the beta testing and I am pretty far along. I have been working with the 1000v team beta testing for about a year and worked on some pretty cool stuff. here is a pic of my rack;

You cant see the dell servers I am using for hosts but they are there in the back. In any case here is a overall look at my lab. I actually had to put 2 new 20 amp circuits in my breaker box to power this stuff!

Heres a little more detailed look into the physical router and switch side. I have a ton of routers and switches, call manager, some phones, ASA’s and my connections to the internet, wireless and the outside garage area where my virtual stuff is. There is some old NAC, MARS,IPS and ACS stuff as well as WAAS mobile and call manager 8. I haven’t played with this stuff in over a year I am really focused on the virtual world

On the virtual side of the house we start with a Netapp 3200 SAN connected via a MDS and fiber channel to my 3 dell servers. I boot ESXi5 from fiber channel and my datastores are NFS and it works perfect. Of course I run vCenter to manage it all. In my virtual I have a domain controller to run the windows environment and the CIAC and cloupia environment. I can deploy VM’s from either CIAC or Cloupia quite easily and both work real well. Since I am studying for the CCIE datacenter test, I have tried to get as many things running to simulate a virtual CCIE lab environment. I have 2 NEXUS 7k simulators running OTV, I have quite a bit of different flavors of the 1000v running, and I have UCS simulators running in there. Also have the MDS storage so I can practice zoning and such. The only thing I really cant do is do a multi-hop FCoE to a storage VDC which looks like that on the lab. This is pretty good for playing around but I will have to rent some rack time once the vendors get their books and labs out with rack rentals. No big rush figuring sometime next year I will make a attempt at the datacenter lab. I also have vWAAS and now the 1000v on Hyper-V so I have quite a bit to play with. Honestly I don’t watch TV I just have fun labbing stuff up. Thankfully my wife is understanding.

NEXUS 7K and Nortel vPC and SMLT interoperatability

We have some customers that are interested in migrating from the Nortel 8600 line over to the NEXUS switches. These are very large clients with mission critical networks so they cannot be taken down and the migration process must be done during small windows with minimal outages. So the basic idea will be as follows;

Stand up the entire infrastructure and connect to the Nortels. The question was would we set it up; using just spanning tree, using SMLT,vPC? In doing a lot of research, when you turn on SMLT it disables spanning tree on the SMLT ports. How would this affect spanning tree on the 7K? Our original proposal to these customers was to use SMLT from the 8600’s and regular spanning tree on the 7K’s as edge switches.

So the next step was that after the switches are in was to migrate the edge switches over to the NEXUS 7k’s via vPC’s and use the 7K’s as distribution switches during this process. All L3 traffic will still be switched on the 8600’s. One client is also replacing their closets switches and going to 3750X stacks so that will be fairly easy. The other client has many Nortel edge switches and these will remain in place. We will simply migrate the port channels and place them in a vPC.

Once the edge and closet switches have been migrated, we will move the datacenter servers onto the new 5k’s and 2248’s. Again this is all layer 2 and all layer 3 switching will be done on the 8600.

The next step was to migrate the layer 3 switching from the 8600’s down to the 7k’s. This would be done by creating P-P SVI’s are using these SVI’s to create OSPF neigbors to get the routing of the 8600’s onto the 7K’s. Once full layer 3 routing was obtained we would simply add the layer 3 interfaces of each subnet to the vrrp groups and once vrrp was up and running we would migrate each subnet from the 8600’s to the 7K’s by simply changing priority and the vrrp default gateway would be moved to the 7K’s 

NEXUS 7k vPC and 3750X stacks vPC failure on reboot.

 I a configuring a customers datacenter infrastructure and during failover testing of the 3750 stacks I get a very unusual error. When the stack fails, it appears to segment the stack and the NEXUS vPC see this as coming from 2 seperate switches. Now the weird thing was that the port channel stays up, however the vPC stays down. Once the switch comes back up all is well however in a failure or power loss scenario of the port channel members this would sever all connectivity of clients on the stack.

2011 Mar 25 23:16:41.277 ODFL-N7010A %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs
1,8,12,16,20,28,32,36,40,44,48,52,56,60,90-91,98-100,108,112,116,120,137-138,152
,200,224,232,252-255,4088-4090,4093 on Interface port-channel8 are being suspend
ed. (Reason: vpc port channel mis-config due to vpc links in the 2 switches

I had followed the vPC config guide and setup all parameters as indicated and the 7K is setup as LACP active and the 3750 stack as LACP passive, with spanning guard root on as well. A quick call to TAC indicated its a know issue with LACP and the stacks, setting the port channels to on with no LACP and removign the root gurad did the trick

Why the title

I tried to come up with something that describes what I work with on a daily basis. Basically I am a network engineer and I deal with taking data from one computer, putting it on a wire and sending it to another computer. If you break it down everything on a computer is based in binary(0’s and 1’s) bits. You take these bits off a hard drive, the OS does its thing and these bits are converted to binary electrical impulses(0’s and 1’s) and sent across the wire to the other computer. So basically these binary bits are taking a journey from one computer to another.

Forgetfulness

So the reason I started doing a blog is I am forgetting a lot of what I do on a daily basis and what I have done in the past. Hopefully in all this writing I can document all the cool things I have done with my life and hopefully help some people with all the mistakes I have made. So welcome to my blog hopefully I can help someone out. Please feel free to contact me and if anything that I blog about is incorrect feel free to tell me about it I try my best to put good information and experiences.