Timeline of the outage: (times in UTC)
Starting at 02:03 on Saturday, requests started intermittently returning error 502, and many of the requests that were served successfully were significantly slower than normal
At 05:59, most of the services on the server crashed. All subsequent requests were served by Cloudflare with error code 502
At about 11:00, I came online and attempted to diagnose the problem. Due to the previous failures I was unable to access the server, so all I was able to do was route traffic to an offline page
At 16:35, with extra help, the server was forcibly restarted
At 16:43 the server seemed to have come up successfully, so we enabled traffic and monitored the status
By 16:53 it was clear that performance very poor and a significant fraction of the requests were resulting in error 502 (which means in this case that the server was already processing too many requests, so where were no workers available), so we disabled traffic again to investigate the situation further
At 17:36, there was nothing clearly wrong so we tried enabling traffic again
At 17:39 the performance was significantly regressing again so the site was turned offline again
At 18:00 we ordered a new server
At 18:35 the server was ready to be set up with all of the forum-related things, and for the data to be migrated to it
At 21:28, the new server was fully set up and the forum was turned back online
What were the symptoms?
IOWait accounted for the majority of CPU time, but IO utilisation was relatively normal
In the syslog we were seeing a number of IO timeouts for the primary drive
Prior to rebooting, the limited errors that we could see indicated that there had been disk corruption
What was the root cause?
Although we aren't 100% sure, we think it's fairly likely that one of the two RAID 1 disks that form the primary disk had failed, and the poor performance was a consequence of trying to rebuild the array.
Why replace the server rather than the disk?
There were already plans to replace the server, this failure just accelerated them. We were also of the opinion that having the disk replaced and getting the array rebuilt would likely not end up being faster, especially as we did not have sufficient information to pinpoint the failure.
If you are aware of updated information regarding telemetry and data collection in Windows 7 / 8 / 10, or know improved methods to disable it, please share it.
For Windows 10
General Privacy Guide's for Windows 10 version 1803 and 1809
This guide includes instructions on how to disable or remove various invasive or unwanted aspects of Windows 10. Some of what the guide includes instructions for:
- Basic Windows 10 set-up choices
- How to remove various apps, such as Windows Store, and other native Win 10 apps
- How to disable Cortana via registry
- How to remove various telemetry services and scheduled tasks
Some of what this guide shows instructions for may already be done by programs such as O&O ShutUp10.
For 1803: https://fdossena.com/?p=w10debotnet/index_1803.frag
For 1809: https://fdossena.com/?p=w10debotnet/index_1809.frag
For 1903: https://fdossena.com/?p=w10debotnet/index_1903.frag
Using an edition of Windows 10 that lets you set the telemetry as low as possible
Windows 10 Enterprise and LTSC allow you to use the group policy editor to lower the amount of data-harvesting to Security Only. The Security Only setting may appear in Windows 10 Pro, but setting the group policy to that setting in Pro doesn't have any effect as the Telemetry Only setting is disconnected from any functionality in Pro.
Licenses for Windows 10 Enterprise and LTSC can be purchased for cheap off of eBay.
To set data-collection to Security Only in Windows 10 Enterprise and LTSC:
1. Open the group policy editor
2. Navigate to: Computer Configuration -> Administrative Templates -> Windows Components -> Data Collection and Preview Builds
3. From the list of that sections policies, double-click on the policy titled Allow Telemetry
4. Set the policy to Enabled, and then set the policy to Security Only from the drop-down box
5. Click OK to close the window
O&O ShutUp10
O&O ShutUp10 is an excellent free piece of software that provides many options to reduce the amount of data that is harvested by Microsoft. I strongly recommend using it and reading the description of each item that can be enabled or disabled to stop a lot of unwanted data-harvesting.
Setting up a custom firewall to block Microsoft telemetry servers
Download and install this custom hosts file, and this custom PeerBlock Microsoft IP list which is regularly updated from information obtained via Wireshark: https://encrypt-the-planet.com/windows-10-anti-spy-host-file/
Review this thorough guide (a website account is needed to view it): https://encrypt-the-planet.com/completely-disable-windows-10-telemetry/
Spybot Anti-Beacon
Another good tool for blocking unwanted communication between a PC and Microsoft. It adds a lot of IPs to the Windows hosts file
Use the free Debloat Windows 10 script to do as it says in its description:
Debloat Windows 10: https://github.com/W4RH4WK/Debloat-Windows-10/blob/master/scripts/block-telemetry.ps1
Manually block Microsoft data-collection servers in your Windows hosts file
There have been suggestions that Windows 10 ignores Microsoft servers in the hosts file, but Spybot Anti-Beacon adds a bunch of Microsoft servers to it and so maybe they know something different.
The Windows hosts file is located at C:\Windows\System32\drivers\etc. To open it, right-click and select "Open with", then choose Notepad and press OK. Then save the file when you're done editing it and then close it.
You can try adding these Microsoft data-collection servers to your Windows hosts file:
More details on Windows 10 endpoints and ways to disconnect them: https://docs.microsoft.com/en-us/windows/privacy/manage-windows-endpoints
My Digital Life's excellent repository on Windows 10 telemetry and its removal that contains sections on:
Delete Telemetry Services
Delete Remnants of Diagtrack and Cortana
Task Scheduler Block
IP Re-Routing
Hosts File Block
Packages Uninstall Lists
PEERBLOCK for Blocking Telemetry
Windows 10 IP Range Block List
Apps Online Uninstall
Link: [REPO] Windows 10 TELEMETRY REPOSITORY
Disable Windows 10 Telemetry Service
Disclaimer: This suggestion has been reported to be deprecated and so likely has no effect on the amount of telemetry Windows 10 collects
It was previously suggested that some Windows 10 telemetry and data collection could be disabled by doing the following:
Go to Services and Applications -> Services in the left pane. In the services list, disable the following service:
Connected User Experiences and Telemetry service (called "Diagnostics Tracking Service" in Windows 10 version 1151 and earlier)
dmwappushsvc
Again, doing this likely has no effect on the level of telemetry that is collected.
For Windows 7 and 8
The surest way to run Windows 7 telemetry-free is to install Windows 7 from an early-to-mid 2015 ISO and then permanently disable Windows Update.
I have a June 2015 ISO of Windows 7 available for download in this post:
Block Microsoft data-collection servers in your Windows hosts file
For this, follow the same instructions mentioned in the Windows 10 section of this guide.
Windows 7 / 8(.1): Guidance on avoiding telemetry-containing updates including a list of security-only updates with download links for each (updated February 2019):
Windows 7 / 8(.1): A thorough guide to identifying telemetry updates, removing the Windows telemetry service, and blocking Microsoft's telemetry servers
Aegis script: Disables, uninstalls, and blocks a bunch of Windows 7 / 8 telemetry & data-collection updates
Since around mid-2015 (and possibly a bit earlier), Microsoft has been back-adding telemetry harvesting into Windows 7 and 8(.1) through the updates that are installed by Windows Update.
These updates are not required, and can be declined from installing and also will not be downloaded and installed if Windows Update is disabled. But, if a person has automatic updates enabled these updates will be automatically installed. If a person doesn't want these to be installed, they can be quickly uninstalled and blocked from being re-downloaded and installed by running a script that knows which updates to permanently block from being searched-for by Windows Update, and from being downloaded and installed.
Aegis Script is one such script, though it was last updated May 18, 2016.
The Aegis script will also remove and block any prompts in Windows 7 and 8 about downloading and installing Windows 10.
Original script link, and discussion: Script for Win 7/8 to block all telemetry updates and Windows 10 upgrade components
Direct-download backup link for Aegis v1.18: https://mega.nz/#!dhExAbBa!fehYhbTNz5dIBh72psfXLfwv9wMk0uhMpGli-c0pBn4
For a list of the Windows 7 and 8(.1) updates that the Aegis script addresses, and also of the post Aegis scrip Windows updates to avoid because they have telemetry in them, see the bottom of this post.
For identifying data-collection-containing updates since 2016, here is an updated list of which updates to avoid or uninstall: https://pastebin.com/jWX2zHdr
------------------------------------------------
Overview of what the Aegis script does
Direct-download backup link for Aegis v1.18: https://mega.nz/#!dhExAbBa!fehYhbTNz5dIBh72psfXLfwv9wMk0uhMpGli-c0pBn4
(This information was retrieved from an archived version of the voat.co page discussing Aegis, via http://pastebin.com/1Xb2h39Z, and was last updated March 27, 2016, and so will not mention any changes to the script after that time)
Description: Blocks 201 bad hosts, change windows update to check/notify (do not download/install), disable automatic delivery of internet explorer via windows update, disable ceip/gwx/skydrive(aka onedrive)/spynet/telemetry/wifisense, disable remote registry, disable 31 scheduled tasks, disable windows 10 download directory, remove diagtrack, sync time to ntp.org, hide/uninstall 50 kb updates (see below).
Directions: Download, unzip, disable anti-virus, right click on aegis.cmd, click "run as administrator", follow on-screen instructions.
Note: If unable to uninstall some kb's this post may help.
Internet Explorer:
Some updates which may contain critical security patches for ie, as well as automated delivery of ie and related updates, will be blocked. Due to the obvious security risk posed by running an unpatched browser we strongly advise to uninstall ie. If you plan to continue to use ie you should probably not run this script - or manually patch and do so at your own risk.
Liability:
All code except sed and setacl is provided as open source so you can look and see for yourself what it does. It has been thoroughly tested on my own systems and scanned with VirusTotal, and to the best of my knowledge it does not contain any harmful or malicious elements. However I assume no liability for any problems so use it at your own risk.
License:
There is no official license - you are welcome to modify and share my code and you do not have to give me credit. I do appreciate any feedback and I will give you credit if I use your ideas. This script is the product of a collaborative effort and does not belong to any one person.
Windows Update:
This script will not block Windows Update however it will change your Windows Update settings to 'check/notify but do not download/install'. If you have problems getting Windows Update to work properly after running the script you may need to run the Windows Update Troubleshooter or the System Update Readiness Tool. If you have recently installed updates and have not yet rebooted you should reboot before running the script. If you are on a fresh install you may want to install all updates before running Aegis for the first time, otherwise it may take a long time to update.
Here is a possibly-incomplete listing of updates that Aegis removes and blocks, using the format:
kb update ID
update description
kb971033
update for windows activation technologies
kb2882822
update for adding itracerelogger interface support
kb2902907
description not available, update was pulled by microsoft
kb2922324
description not available, update was pulled by microsoft
kb2952664
update for upgrading windows 7
Ugh, screw this.
[kb2976978](https://support.microsoft.com/en-us/kb/2976978) | update for windows 8.1 and windows 8
[kb2977759](https://support.microsoft.com/en-us/kb/2977759) | update for windows 7 rtm
[kb2990214](https://support.microsoft.com/en-us/kb/2990214) | update that enables you to upgrade from windows 7 to a later version of windows
[kb3012973](https://support.microsoft.com/en-us/kb/3012973) | upgrade to windows 10
[kb3014460](https://support.microsoft.com/en-us/kb/3014460) | update for windows insider preview / upgrade to windows 10
[kb3015249](https://support.microsoft.com/en-us/kb/3015249) | update that adds telemetry points to consent.exe in Windows 8.1 and Windows 7
[kb3021917](https://support.microsoft.com/en-us/kb/3021917) | update for windows 7 sp1 for performance improvements
[kb3022345](https://support.microsoft.com/en-us/kb/3022345) | update for customer experience and diagnostic telemetry
[kb3035583](https://support.microsoft.com/en-us/kb/3035583) | update installs get windows 10 app in windows 8.1 and windows 7 sp1
[kb3042058](https://support.microsoft.com/en-us/kb/3042058) | update for cipher suite priority order (contains winlogon spying elements)
[kb3044374](https://support.microsoft.com/en-us/kb/3044374) | update that enables you to upgrade from windows 8.1 to windows 10
[kb3046480](https://support.microsoft.com/en-us/kb/3046480) | update for migrating .net when upgrading to later version of windows
[kb3058168](https://support.microsoft.com/en-us/kb/3058168) | activate windows 10 from windows 8 or windows 8.1, and windows server 2012 or windows server 2012 r2 kms hosts
[kb3064683](https://support.microsoft.com/en-us/kb/3064683) | update for windows 8.1 oobe modifications to reserve windows 10
[kb3065987](https://support.microsoft.com/en-us/kb/3065987) | update for windows update client for windows 7 and windows server 2008 r2 july 2015
[kb3065988](https://support.microsoft.com/en-us/kb/3065988) | update for windows update client for windows 8.1 and windows server 2012 r2 july 2015
[kb3068708](https://support.microsoft.com/en-us/kb/3068708) | update for customer experience and diagnostic telemetry
[kb3072318](https://support.microsoft.com/en-us/kb/3072318) | update for windows 8.1 oobe modifications to reserve windows 10
[kb3074677](https://support.microsoft.com/en-us/kb/3074677) | compatibility update for upgrading to windows 10
[kb3075249](https://support.microsoft.com/en-us/kb/3075249) | update that adds telemetry points to consent.exe in windows 8.1 and windows 7
[kb3075851](https://support.microsoft.com/en-us/kb/3075851) | update for windows update client for windows 7 and windows server 2008 r2 august 2015
[kb3075853](https://support.microsoft.com/en-us/kb/3075853) | update for windows update client for windows 8.1 and windows server 2012 r2 august 2015
[kb3080149](https://support.microsoft.com/en-us/kb/3080149) | update for customer experience and diagnostic telemetry
[kb3081437](https://support.microsoft.com/en-us/kb/3081437) | august 18, 2015, compatibility update for upgrading to windows 10
[kb3081454](https://support.microsoft.com/en-us/kb/3081454) | september 8, 2015, compatibility update for upgrading to windows 10
[kb3081954](https://support.microsoft.com/en-us/kb/3081954) | update for work folders improvements in windows 7 sp1 (contains telemetry elements)
[kb3083324](https://support.microsoft.com/en-us/kb/3083324) | update for windows update client for windows 7 and windows server 2008 r2 september 2015
[kb3083325](https://support.microsoft.com/en-us/kb/3083325) | update for windows update client for windows 8.1 and windows server 2012 r2 september 2015
[kb3083710](https://support.microsoft.com/en-us/kb/3083710) | update for windows update client for windows 7 and windows server 2008 r2 october 2015
[kb3083711](https://support.microsoft.com/en-us/kb/3083711) | update for windows update client for windows 8.1 and windows server 2012 r2 october 2015
[kb3086255](https://support.microsoft.com/en-us/kb/3086255) | september 8, 2015, security update for the graphics component in windows (breaks safedisc)
[kb3088195](https://support.microsoft.com/en-us/kb/3088195) | october 13, 2015, security update for windows kernel (reported to contain a keylogger)
[kb3090045](https://support.microsoft.com/en-us/kb/3090045) | windows update for reserved devices in windows 8.1 or windows 7 sp1 (windows 10 upgrade elements)
[kb3093983](https://support.microsoft.com/en-us/kb/3093983) | security update for internet explorer: october 13, 2015 (ie spying elements)
[kb3102810](https://support.microsoft.com/en-us/kb/3102810) | windows 10 upgrade elements
[kb3102812](https://support.microsoft.com/en-us/kb/3102812) | windows 10 upgrade elements
[kb3112343](https://support.microsoft.com/en-us/kb/3112343) | update for windows update client for windows 7 and windows server 2008 r2 december 2015
[kb3112336](https://support.microsoft.com/en-us/kb/3112336) | update for windows update client for windows 8.1 and windows server 2012 r2 december 2015
[kb3123862](https://support.microsoft.com/en-us/kb/3123862) | updated capabilities to upgrade windows 8.1 and windows 7
[kb3135445](https://support.microsoft.com/en-us/kb/3135445) | windows update client for windows 7 and windows server 2008 r2: february 2016
[kb3135449](https://support.microsoft.com/en-us/kb/3135449) | windows update client for windows 8.1 and windows server 2012 r2: february 2016
[kb3138612](https://support.microsoft.com/en-us/kb/3138612) | windows update client for windows 7 and windows server 2008 r2: march 2016
[kb3138615](https://support.microsoft.com/en-us/kb/3138615) | windows update client for windows 8.1 and windows server 2012 r2: march 2016
[kb3139929](https://support.microsoft.com/en-us/kb/3139929) | security update for internet explorer: march 8, 2016
[kb3146449](https://support.microsoft.com/en-us/kb/3146449) | updated internet explorer 11 capabilities to upgrade windows 8.1 and windows 7
Updates including post-Aegis that contain telemetry, updated to June 2018:
Windows 7/8/8.1 Updates to avoid as of the June 2018 "Patch Tuesday":
KB971033, Activation exploits
KB2876229, Skype
KB2882822, replaced by KB3068708
KB2952664, telemetry crap
KB2970228, new Russian ruble symbol, breaks fonts
KB2976978, Windows 10 update crap for Win8
KB2977759, telemetry crap
KB2982791, Causes crashes
KB2990214, telemetry crap
KB3004394, faulty update
KB3018238, only applies to Windows Server 2008
KB3021917, telemetry crap
KB3022345, telemetry crap
KB3035583, telemetry crap
KB3050265, telemetry crap
KB3065987, telemetry crap
KB3068708, telemetry crap
KB3075249, telemetry crap
KB3075851, telemetry crap
KB3080149, telemetry crap
KB3081954, telemetry crap
KB3083324, telemetry crap
KB3083710, telemetry crap
KB3097877, Casuses crashes
KB3102810, telemetry crap
KB3107998, Lenovo fix to remove blocker
KB3112336, More WIN10 crap
KB3112343, More WIN10 crap + MS monitoring of win10 upgrade
KB3121255, crash during backup of PI Data server fails
KB3123862, Windows 10 update crap
KB3125574, Apr 2016 rollup with bad ones in it
KB3133977, BitLocker can't encrypt the drive and the service crashes
KB3135445, WIN7 update client to force WIN10
KB3137061, Azure virtual machines network outage data corruption
KB3138901, No Internet multiple users log on Remote Desktop Services
KB3139923, MSI repair doesn't work after you install updates
KB3147071, Connection to Oracle database fails. Causes browser lockups?
KB3150513, telemetry crap
other:
KB3184143 removes the Get Windows 10 app
KB3172605 July 2016 update rollup (re-released Sep 13 2016)
KB3179573 August 2016 Rollup
myselfolli reacted to colonel_mortis for a blog entry, A breakdown of Saturday's outage
