Locate network host...

[Tzomb1e]

I apologize if this belongs in another section...due to the issues, networking seemed most appropriate. 

 

I am currently searching for a couple unknown hosts on a network that consists of more than 60 subnets with almost as many physical locations.  All remote locations connect back to the core with MPLS.  Normally, I can follow the L3 or L2 information back through the infrastructure to discover the source.  However, the hosts in question are using a network that does not exist within the architecture and can only be seen sending traffic to addresses that are very similar to my DNS servers.  By similar, I mean they share the same last two octets, but not the first two.  I was able to create routes to direct the DNS traffic towards a sink hole in order to capture the packets and see the contents of the requests...but this only yielded queries for windows time servers, NCSI servers, and other windows based system services without any host information.  Since there is no preexisting network route (static or shared), there is no trail to follow back to the host.  Any L2 information in the captured packets only show the MAC from the forwarding gateway, as expected from L2 logic.  Unless my brain is misfiring, the host would have to be configured with a gateway that falls within it's IP subnet (especially for a windows host) in order to properly forward any non-lan requests to external networks.  Since the network is not one that "exists" in the architecture, this leads to a bit of assumption that another router would have to be in place (not using NAT unless dynamic is being used since I have two different source addresses) in order for the hosts to send the traffic to its gateway, which can then send the traffic into my other networks assuming next hop information has been added (or the routing device is pulling a DHCP lease). 

 

Creating SPAN ports on the core infrastructure devices is not a possibility due to even the slightest risk of network degradation.  A tap is possible, but I think I would end up with the same information from my pseudo honeypot/sinkhole. 

 

Anyone have any advice that would save me from touching every device at all 60 locations?  I am going to dig through the leases to see if any hostnames might stand out, but this is still tedious.  While I realize there are a million technologies for asset management that would assist with this, the company in question has never really cared about proper infrastructure maintenance and monitoring...which is why this is being done by hand. 

[Windows7ge]

I wish I had even a fraction of the knowledge that would go into setting up something this complex.

 

Here are some individuals who may be able to help (in no particular order):

@leadeater

@mynameisjuan

@Lurick

@Mikensan

[Mikensan]

You should be able to pull up the ARP tables in your switches, dump it to a file. Then dump the MAC addresses from your DHCP / IPAM - compare the two. 

 

I only read your post partially - if it's getting a DHCP IP address then it's in a valid subnet and you can just use a tool like nmap or zenmap (windows) and scan your ranges in bulk. This is assuming their device will reply to a ping for quick scans. 

 

PDQ inventory has a 14 day trial that might help as well.

[Tzomb1e]

8 hours ago, Windows7ge said:

I wish I had even a fraction of the knowledge that would go into setting up something this complex.

 

Here are some individuals who may be able to help (in no particular order):

@leadeater

@mynameisjuan

@Lurick

@Mikensan

Thank you for the recommendations!

3 minutes ago, Mikensan said:

You should be able to pull up the ARP tables in your switches, dump it to a file. Then dump the MAC addresses from your DHCP / IPAM - compare the two. 

That is one of the options I have been debating...since I cannot pin down the exact location that the traffic is coming from, it would involve more than 100 switches :D.  I realize I am a bit limited on my options to begin with, but I just wanted to make sure I was not missing something simple.  I appreciate the reply!

[Mikensan]

Just now, Tzomb1e said:

Thank you for the recommendations!

That is one of the options I have been debating...since I cannot pin down the exact location that the traffic is coming from, it would involve more than 100 switches :D.  I realize I am a bit limited on my options to begin with, but I just wanted to make sure I was not missing something simple.  I appreciate the reply!

You can create a script using SSH to cycle through all 100 switches assuming they're all the same brand. Might be a good time to setup a syslog server for future endeavours too (tweak earlier script to cycle through them all and set the syslog server). There are configuration management tools for almost every switch out there, helps you push configs to them all at once.

 

A longer term solution might be some type of network access control which is honestly a nightmare and a full time job.

 

If they're not using your gateway(s) then your only source of information are your switches, though tidious you may be powering through them all. Just hopefully they're all the same brand.

[Mikensan]

Oh another thought, if you are going through your DHCP logs and assuming all 60 locations use the same brand of endpoints - sort by MAC address. Unless they're also using the same brand and generation as your endpoints, which makes life difficult.

 

Some DHCP servers allow you to only give IP addresses based on information from the requesting device - you might be able to fine tune it and shorten the leases. For example you can say only Windows 10 devices may get an IP etc... Have to definitely be careful here though lol.

 

Are you doing this out of curosity or has something been brought to your attention that makes you think there are some rogue devices?

[Tzomb1e]

Just now, Mikensan said:

You can create a script using SSH to cycle through all 100 switches assuming they're all the same brand. Might be a good time to setup a syslog server for future endeavours too (tweak earlier script to cycle through them all and set the syslog server). There are configuration management tools for almost every switch out there, helps you push configs to them all at once.

 

A longer term solution might be some type of network access control which is honestly a nightmare and a full time job.

 

If they're not using your gateway(s) then your only source of information are your switches, though tidious you may be powering through them all. Just hopefully they're all the same brand.

I did not consider a script, for some silly reason.  I will work on that.  Thanks!

 

I have been trying to convince those that make choices to allow my team to push forward with 802.1x or ISE or really any NAC...but all they see is cost...and, like you said, the administrative overhead would lead to a larger headcount. 

 

Thankfully we do use the same vendor for our infrastructure equipment.  They have not been properly cared for, however, and centralized management/logging has been hit or miss during setups. 

[Tzomb1e]

Just now, Mikensan said:

Oh another thought, if you are going through your DHCP logs and assuming all 60 locations use the same brand of endpoints - sort by MAC address. Unless they're also using the same brand and generation as your endpoints, which makes life difficult.

 

Some DHCP servers allow you to only give IP addresses based on information from the requesting device - you might be able to fine tune it and shorten the leases. For example you can say only Windows 10 devices may get an IP etc... Have to definitely be careful here though lol.

 

Are you doing this out of curosity or has something been brought to your attention that makes you think there are some rogue devices?

I have considered this, but the corporate policies are somewhat lax on what can be used on the network which makes this way too tricky for me to truly consider.  Some of the scope options, like lease, I have already worked on. 

 

The devices in question were detected by our IPS/IDS and I have been monitoring their traffic across our edge firewall.  While I can just block them logically, finding them physically would be best.  With my lack of resources, I may just have to settle for the former...but one can dream!

[Mikensan]

1 minute ago, Tzomb1e said:

I did not consider a script, for some silly reason.  I will work on that.  Thanks!

 

I have been trying to convince those that make choices to allow my team to push forward with 802.1x or ISE or really any NAC...but all they see is cost...and, like you said, the administrative overhead would lead to a larger headcount. 

 

Thankfully we do use the same vendor for our infrastructure equipment.  They have not been properly cared for, however, and centralized management/logging has been hit or miss during setups. 

Hopefully you get one working, I'm sure somebody out-there already has one that just needs a little tweaking lol.

ISE would be great, but definitely adds cost for sure. That initial rollout and verifying devices gets annoying too. Just have to weigh the value of insider threat protection.

Thankfully information digging only requires very basic CLI commands which almost never change. So, even if they aren't all the same version, your script should work. If they buck at NAC I would definitely push hard for centralized logging.

[Mikensan]

1 minute ago, Tzomb1e said:

I have considered this, but the corporate policies are somewhat lax on what can be used on the network which makes this way too tricky for me to truly consider.  Some of the scope options, like lease, I have already worked on. 

 

The devices in question were detected by our IPS/IDS and I have been monitoring their traffic across our edge firewall.  While I can just block them logically, finding them physically would be best.  With my lack of resources, I may just have to settle for the former...but one can dream!

Oh cool, so you should definitely have their IP / MAC at hand? Not sure of your IDS/IPS configuration, but if it is inline then abosutely these devices are passing through a gateway. But if they're working of mirrored ports / flow data, then maybe not. Is it possible to increase the verbosity in the logs for a short window of time when you expect these devices to connect?

[Tzomb1e]

Just now, Mikensan said:

Oh cool, so you should definitely have their IP / MAC at hand? Not sure of your IDS/IPS configuration, but if it is inline then abosutely these devices are passing through a gateway. But if they're working of mirrored ports / flow data, then maybe not. Is it possible to increase the verbosity in the logs for a short window of time when you expect these devices to connect?

I have the IPs, but the only MAC is from the forwarding gateway...which is the forwarding interface of our core router :D.  Since it appears to be in a different broadcast domain, I lose the original MAC the second it is forwarded to another LAN segment.  The traffic, at least from what the IDS/IPS, firewall, and my sinkhole has caught, has only been DNS queries that are not registering the device name with our DHCP.  So the only information the logs give me is the RFC1918 address of the device and the site it is trying to resolve (which has all been normal windows traffic for Microsoft time and NSCI servers).  The IP is almost useless since the subnet it is using does not exist in my environment. 

[Mikensan]

16 minutes ago, Tzomb1e said:

I have the IPs, but the only MAC is from the forwarding gateway...which is the forwarding interface of our core router :D.  Since it appears to be in a different broadcast domain, I lose the original MAC the second it is forwarded to another LAN segment.  The traffic, at least from what the IDS/IPS, firewall, and my sinkhole has caught, has only been DNS queries that are not registering the device name with our DHCP.  So the only information the logs give me is the RFC1918 address of the device and the site it is trying to resolve (which has all been normal windows traffic for Microsoft time and NSCI servers).  The IP is almost useless since the subnet it is using does not exist in my environment. 

does the forwarding gateway have any virtual interfaces / IPs configured by chance?

[Mikensan]

are you free to post the logs minus identifying information?

[Tzomb1e]

15 minutes ago, Mikensan said:

does the forwarding gateway have any virtual interfaces / IPs configured by chance?

The forwarding interface does not have any virtual interfaces.  The ingress interface does, though. 

16 minutes ago, Mikensan said:

are you free to post the logs minus identifying information?

Unfortunately I cannot share any of the logs.  While they do not contain much other than the packet information that I am pulling from my sinkhole, which only has IP, MAC, and information from the DNS query, I am not allowed to share them.  Sorry!

[schizznick]

You state you are using MPLS, I am guessing you are not running the MPLS cloud and are instead the customer? If you are, is it a VPLS, VLL , VPRN or something else?

[Mikensan]

22 hours ago, Tzomb1e said:

The forwarding interface does not have any virtual interfaces.  The ingress interface does, though. 

Unfortunately I cannot share any of the logs.  While they do not contain much other than the packet information that I am pulling from my sinkhole, which only has IP, MAC, and information from the DNS query, I am not allowed to share them.  Sorry!

That's all good, I certainly understand. I work on some air-gapped networks and in the same boat. I love the investigation process, wish I could sit in the same room and track those devices down lol. At least it appears they are windows based,creating lots of noise on the network. No Jr IT people at the remote sites?

[Tzomb1e]

14 hours ago, Mikensan said:

That's all good, I certainly understand. I work on some air-gapped networks and in the same boat. I love the investigation process, wish I could sit in the same room and track those devices down lol. At least it appears they are windows based,creating lots of noise on the network. No Jr IT people at the remote sites?

Investigations of any kind are normally my favorite part of the job, they tend to get passed a little further down the chain anymore however.  This is the first one I have been able to take on myself in a while due to copious other projects.  And that is more or less all it is: a windows device making noise that got my attention.  Our field support team just acquired three greenies right out of school, I have already had my normal "this network is not your playground" talk and specifically asked them about this traffic but I do have my doubts.  Fortunately I have some surprise posture assessments in their territories next week that will let me validate their stories, so we will see how things hold up. 

[leadeater]

On 6/26/2019 at 1:01 AM, Tzomb1e said:

I have been trying to convince those that make choices to allow my team to push forward with 802.1x or ISE or really any NAC...but all they see is cost...and, like you said, the administrative overhead would lead to a larger headcount. 

A well implemented 802.1X doesn't carry much burden at all and would most likely have saved your ass here in terms of time .

 

Centralized syslog server would also help here too, we send our logs to Greylog so they can be fully indexed and searched so getting useful information out is fast and easy.

 

But the above isn't much use now, can't go back in time to implement tools to help.

 

7 hours ago, Tzomb1e said:

surprise posture assessments

What the heck is that?

 

On 6/26/2019 at 1:19 AM, Tzomb1e said:

Since it appears to be in a different broadcast domain, I lose the original MAC the second it is forwarded to another LAN segment.

Shouldn't you be able to follow the path back to each forwarding device checking ARP tables as you go? Seems odd you can't do that or at least be able to narrow down the network segment from that.

 

On 6/26/2019 at 1:19 AM, Tzomb1e said:

The IP is almost useless since the subnet it is using does not exist in my environment. 

Using the information you know about your network and your routing tables is there a more likely segment it would have come from? Would every segment allow this source IP to be routed to this destination or would it end up somewhere else depending on where it originates?

 

Sometimes it's easier to eliminate the impossible options then work from that.

[mynameisjuan]

Sorry for the delay but I just skimmed through the post here,

 

@Tzomb1e

 

You should be implementing DHCP relay with option 82 implemented on the server where every possible and dotx for every static address.

 

Option 82 with relay essentially encapsulated the switch/router TLVs for the device and is sent directly to the server. This is a sudo authentication but most importantly it tells you EXACTLY where the malicious device is

 

Essentially the DHCP will keep a log of all leases with each lease having a binding of the switch/router hostname, IP, vlan and port number. This makes it simple and easy to track down any DHCP client because the lease tells you the exact device and what port its off.

 

As an ISP we by law have to have this level of tracking for legal reasons with customers and verifying authentication and this is by far the easiest way to find a culprit. There are better ways with software but I have yet to go that route. 

 

With static devices, dot1x should be used and tracked, there is no getting around that.