Dell iDRAC 6 Virtual Console Connection Failed

[gabe927]

I have a Dell PowerEdge R710 with an Enterprise iDRAC 6 controller.  I have an issue connecting to the virtual console.  It uses java and I pointed the jnlp file to open with the Java Web Start Launcher as instructed by a quick google search.  Every time I try to run it, I eventually end up with the error shown in the picture.  I have also added the ip address to the exception list in the java control panel and no luck.  I have tried this with both the latest version of Java as well as version 7 on both Mac and Windows and still no luck.  Any ideas for getting java to work or suggestions to other applications that can run the virtual console would be appreciated. 

[xKOSMOS87]

This is what I've done to get the console working on an iDRAC6 in Windows 10

- Add the iDRAC IP address into your Trusted sites as both https and http, and set the security to Low for trusted sites (Internet Options > Security Tab)

- Add the iDRAC to the Compatibility View settings.

- Add the iDRAC into Java security exclusions as https

- Use IE to login to the web console and launch the Virtual Console

[Mikensan]

Are you on the same subnet as the idrac/ipmi controller? Windows Firewall running? Antivirus?

Did you go control control panel > java, and add the ip address under security?

You shouldn't have to try to open it with anything, should just be able to open the file natively.

[gabe927]

36 minutes ago, xKOSMOS87 said:

This is what I've done to get the console working on an iDRAC6 in Windows 10

- Add the iDRAC IP address into your Trusted sites as both https and http, and set the security to Low for trusted sites (Internet Options > Security Tab)

- Add the iDRAC to the Compatibility View settings.

- Add the iDRAC into Java security exclusions as https

- Use IE to login to the web console and launch the Virtual Console

Thanks! I got it working in IE.  Is there any way to do it on Mac?  I don't want to have to boot up a virtual machine just to use IE.

 

34 minutes ago, Mikensan said:

Are you on the same subnet as the idrac/ipmi controller? Windows Firewall running? Antivirus?

Did you go control control panel > java, and add the ip address under security?

You shouldn't have to try to open it with anything, should just be able to open the file natively.

Same subnet: yes

Firewall/Anitvirus: Temp. disabled and exclusions were added. No luck

Java security: as mentioned above, I did that

You do have to open it manually with the web starter because there is extra info after the file name.  Ex. viewer.jnlp(details about the server)

 

Edit: I tried deleting the extra stuff after .jnlp, same error.

[xKOSMOS87]

45 minutes ago, gabe927 said:

Thanks! I got it working in IE.  Is there any way to do it on Mac?  I don't want to have to boot up a virtual machine just to use IE.

For Mac, you may need to enable SSLv3 support for java by editing the /lib/security/java.security in your Java installation directory, and comment out the line "jdk.tls.disabledAlgorithms=SSLv3"

 

I don't recommend that though as it allows any site with SSLv3 to use Java, and SSLv3 is a vulnerable protocol.  Using something like VMware fusion to run the VM in unity mode is the more secure option.

[Mikensan]

12 minutes ago, gabe927 said:

Thanks! I got it working in IE.  Is there any way to do it on Mac?  I don't want to have to boot up a virtual machine just to use IE.

 

Same subnet: yes

Firewall/Anitvirus: Temp. disabled and exclusions were added. No luck

Java security: as mentioned above, I did that

You do have to open it manually with the web starter because there is extra info after the file name.  Ex. viewer.jnlp(details about the server)

 

Edit: I tried deleting the extra stuff after .jnlp, same error.

Ah yea I forgot which of my servers added that crap after the extension (my other is a supermicro board) - yea I just delete it and it ran without issue. I have had problems where I tried to open it with javaw manually even though that's what the default app is. I don't remember what the issue was unfortunately.

 

Life is easier with it on the same subnet at least.

 

It shouldn't matter IE/FF/Chrome/Safari since it's Java that's opening the file. I don't know how or where to on the Mac, but clear out any temporary java apps since you've tried to run this a few times.

 

However I just installed the latest flavor of java, 8u144 and I had to modify java.security in order to get it working - though I received a different error message than you.

 

Under iDRAC6 > Console/Media > Configuration, these are my settings:

Enabled: Yes

Max Sessions: 4

Remote Presence Port 5900

Video En Enabled: Yes

Local Server Vid En: Yes

Plug-in Type: Native

 

[Mikensan]

Theres a monolithic update for iDRAC that seems to sign using SHA-2 instead of MD5 if you go to dell's website. Should make it happy without having to play with java security settings.

[Mikensan]

Can confirm, went from version 1.7 to 2.9, do not need to modify java.security. So give that a shot, maybe your iDRAC needs to be updated.

[gabe927]

3 hours ago, Mikensan said:

Can confirm, went from version 1.7 to 2.9, do not need to modify java.security. So give that a shot, maybe your iDRAC needs to be updated.

I was running 2.85 previously.  Will have to try 2.9 tomorrow when I have the time.

[Mikensan]

2 hours ago, gabe927 said:

I was running 2.85 previously.  Will have to try 2.9 tomorrow when I have the time.

Dang, definitely confusing why it won't run on the MAC then. 2.9 did outline it switches the hash but not much else, so I don't know if it will help but hope it does. Maybe try clearing java temps from your Mac, unfortunately I don't have one to try out or else I would.

[gabe927]

I finally got a chance to update to 2.9.  Still no luck.  I dug a little deeper and realized that I had never updated the ssl certificate for the iDRAC.  Did that, and it still won't work.

[gabe927]

On 8/16/2017 at 4:22 PM, xKOSMOS87 said:

For Mac, you may need to enable SSLv3 support for java by editing the /lib/security/java.security in your Java installation directory, and comment out the line "jdk.tls.disabledAlgorithms=SSLv3"

 

I don't recommend that though as it allows any site with SSLv3 to use Java, and SSLv3 is a vulnerable protocol.  Using something like VMware fusion to run the VM in unity mode is the more secure option.

Just for giggles, I wanted to try and enable sslV3 to see if that would fix it.  I found the java.security file, and that line does not exist in mac.  Checked it in windows, and sure enough it was there.  I tried to enable it on windows and run the java app, but it still wouldn't connect, so I disabled it again.

 

Edit: Found it in mac, was in a completely different directory.  Still didn't work though.

Edited August 18, 2017 by gabe927
Found directory

[Mikensan]

14 minutes ago, gabe927 said:

Just for giggles, I wanted to try and enable sslV3 to see if that would fix it.  I found the java.security file, and that line does not exist in mac.  Checked it in windows, and sure enough it was there.  I tried to enable it on windows and run the java app, but it still wouldn't connect, so I disabled it again.

Regarding your windows that is strange. I'm running Windows 10 8u144, after 2.9 I don't have to. Within iDRAC what is your plugin type set to?

[gabe927]

4 minutes ago, Mikensan said:

Regarding your windows that is strange. I'm running Windows 10 8u144, after 2.9 I don't have to. Within iDRAC what is your plugin type set to?

It is set to native.  Whenever I open it in chrome, it downloads java which is what I need to use to do it nattily in mac.  When I use IE it will default to ActiveX.  

I noticed from videos of people connecting through java that they get a "DRAC Console Redirection Client" window before they get the actual console.  I never get to that point, I just get a "connection failed" window.

[Jarsky]

I had to install this to open the iDrac .jnlp from Chrome: https://chrome.google.com/webstore/detail/fix-idrac-jnlp-file/knpcepbijjjpmlhbpmkjknghbeghiibo?hl=en

It renames the .jnlp file so Chrome doesnt panic. 

[gabe927]

1 minute ago, Jarsky said:

I had to install this to open the iDrac .jnlp from Chrome: https://chrome.google.com/webstore/detail/fix-idrac-jnlp-file/knpcepbijjjpmlhbpmkjknghbeghiibo?hl=en

That just renames the file so that you don't have to use "Open with".  I've been renaming it manually to make it work.  Thanks for posting though, it at least removes an extra step!

[Mikensan]

Mine is currently set to "java(automatically)" after the 2.9 update. On your mac you've tried with safari/firefox/chrome? When you try to open it with your mac, you're getting the dialog box posted in your earlier post? Oh yea is your iDRAC NIC dedicated or shared?

 

[Mikensan]

"I noticed from videos of people connecting through java that they get a "DRAC Console Redirection Client" window before they get the actual console.  I never get to that point, I just get a "connection failed" window."

 

I get a few certificate warnings, downloads the java applet, then I see "connecting to virtual console server" and that's it. I don't think I've noticed the DRAC Console Redirect Client window. I'm running an R610 but I imagine we're pretty 1:1.

 

Firefox behaves beautifully when opening the file, might have to switch to using that for my impi needs (seldom as they are).

[gabe927]

4 minutes ago, Mikensan said:

Mine is currently set to "java(automatically)" after the 2.9 update. On your mac you've tried with safari/firefox/chrome? When you try to open it with your mac, you're getting the dialog box posted in your earlier post? Oh yea is your iDRAC NIC dedicated or shared?

 

I have dried it on all three browsers.  Same dialog.  Dedicated.  

I'm running an R710, so yea we're pretty 1:1.

[Mikensan]

I'm perplexed, as I'm sure you are too. Trying to think what's different between us. My drac is on a different subnet, so if anything I've got an extra layer of complication. As far as you're getting I'm going to assume your Java is working just fine, but whatever IP address it's trying to connect to fails.

 

Does opening the virtual media have the same issue?

 

Do you have IPMI enabled over LAN?

[gabe927]

Just out of curiosity, if you go to idrac settings > network/sequrity > ipmi settings, is your encryption key a large string of 0s?

[Mikensan]

2 minutes ago, gabe927 said:

Just out of curiosity, if you go to idrac settings > network/sequrity > ipmi settings, is your encryption key a large string of 0s?

yup

[Mikensan]

since you're on the same subnet I don't think this changes anything, but did you specify a gateway?

[gabe927]

2 minutes ago, Mikensan said:

since you're on the same subnet I don't think this changes anything, but did you specify a gateway?

yep

[gabe927]

I was about to give up, but over a quick lunch break, I decided to do a factory reset to the DRAC card, go figure, that fixed it. 

Thanks all for the help!