New Blog

Dear reader,

To work more inline with my twitter activities and to open my blog for more interessting stuff (not just Veeam related), I am blogging only on the http://andyandthevms.com/ blog from now.

If I will write some updates at individual posts (e.g. Exchange Backup stuff) I will add an warning message with a link to the article on this blog.

So do not hesitate to check out my new blog under http://andyandthevms.com

Thanks for all the Feedback and support

Andy

Sonntag, 11. Mai 2014

No SSO visible at vSphere Web Client 5.5 (Reset vSphere Administrator password because of "!" in the password)

Hi,

maybe this is an well known thing, but I never used an "!"  at vSphere Password before and experienced some "inconvenience". I installed vCenter Server and was not able to see the vcenter and also not able to access SSO configuration, because it was not there. (Remeber SSO configuration in 5.5 was rewritten and it is normaly found directly in vSphere Web Client - Administration tab.

"!" are not an allowed character at vSphere SSO Administrator password, but Setup process allow it.
If you used it in you password, you are able to logon to Web Client but you see no vSphere Server nor are you able to see the SSO configuration area. If you do not want to install your Server from scratch, you can use the following command to change the password:

Windows:
cmd
c:\Program Files\VMware\Infrastructure\VMware\CIS\vmdird\vdcadmintool.exe
3
cn=Administrator,cn=users,dc=vSphere,dc=local

Appliance:
Open SSH connection
/usr/lib/vmware-vmdir/bin/vdcadmintool
3
cn=Administrator,cn=users,dc=vSphere,dc=local


The funny thing is, that you need to take care as well, that there is not an "!" in the auto generated password ;o)

After that login to Web Client and you can access SSO configuration now. In my case the vSphere Server showed up automatically after this as well.



Mittwoch, 26. März 2014

Recommended Hyper-V Patches (and Failover Cluster Patches)

Hi everybody,

in the last time more and more customers test Hyper-V and migrate their workloads.
Many of them are not aware, that Hyper-V need some critical updates for a stable operation (and Backup).

Please check the following Links:

Veeam list of all patches that are recommended before doing backups (independent which backup software you use):


And here you can find the links to the Microsoft recommended patches.
If you use CSV,CSVv2 or SMB3 Shared Volumes check also the second link of each OS Version.

Win2012 R2
Win2012 R2 (Hyper-V) Failover Cluster

http://support.microsoft.com/kb/2920151/de


Win2012 Hyper-V
Win2012 (Hyper-V) Failover Cluster


Win2008R2 SP1 Hyper-V
Win2008R2 SP1 (Hyper-V) Failover Cluster


Win2008R2 Hyper-V
Win2008R2 (Hyper-V) Failover Cluster

Montag, 10. März 2014

Exchange (DAG) VMware Backups: Updated list of tips and tricks for Veeam Backup & Replication

This is an old article. You can find a newer version with Update 5 and beyond at:
http://andyandthevms.com/exchange-dag-vmware-backups-updated-list-of-tips-and-tricks-for-veeam-backup-replication/


Update 1: Added Exchange & Virtualization + NFS Support statement. Split of DAG failover / 20 sec timeout tips.
Update 2: At 1.4. blogger completely destroyed my article as a April-Joke, so I had to rearrange the text again. I added some small updates to the text based on your feedback. Thanks for sending me change ideas.
Update 3: 11.April 2014 Added some planning informations round about CPU/RAM/DISK to section 2. and 3.
Update 4: 24. April 2014 Added Exchange Update Link with Exchange 2013 CU3 example.

Here are my updated general recommendations for Exchange/Exchange DAG at VMware together with Veeam Backup & Replication.


1. Check Microsoft Exchange 2013 Virtualization Topic:

http://technet.microsoft.com/en-us/library/jj619301.aspx
- Exchange 2013 DAG are supported on virtualization platforms.
- Microsoft say: Snapshots are not supported, because they are not application aware and can have unintended and unexpected consequences.

Because Microsoft do not support Snapshots at Exchange themselves with, you need to contact your backup or virtualization vendor for any snapshot related questions/support requests. So why using a Snapshot based Backup instead of a Agent based backup method? You can easily restore servers in a minimum time (Instant VM Recovery can bring back the Server in a minute + boot time), Backup time window can be achieved with virtualization backup more easily (so you can perform backups more often on exchange full level), in case of Veeam it will support any new Patch/Service Release by design without an Veeam own Patch... and with Veeams "Veeam Explorer for Exchange" you can restore Exchange objects very fast and uncomplicated. Compared to many others Exchange object restore backup software, Veeam is in many cases very affordable.


However if you use Virtualization based backups (VMware/Hyper-V) you need snapshots for backup. In case of Veeam by enabling Veeam Guest Processing, Veeam uses VSS to bring Exchange in an Application aware backup/restore state before snapshot are started. You can check this by Windows Event Log. There you can find Messages that say "Exchange VSS Writer ((instance GUID) has processed pre-restore events successfully." and "Exchange VSS Writer (instance GUID) has processed post-restore events successfully.". The Event IDs are around number 96xx. They are different in each Exchange Version, so search for the text to check if your Backup application use VSS on a Microsoft recommended way.

2. NFS Datastores are not supported

http://windowsitpro.com/blog/nfs-and-exchange-not-good-combination
Please check out this article, it will help to understand why NFS Datastores for Exchange loads are not an good idea.  (Teasers: Exchange Event ID 1018; NFS can abort transactions only on a best effort base and can cause corruptions in the database.)

However Microsoft added neccessary functionallity to SMB3.0, so in case you run Exchange on  Hyper-V Win2012/2012R2 you can use SMB3.0 shared storage on Windows File Server 2012/2012R2. You have to place Exchange data into VHD/VHDX then.

3. VMware+Exchange Design + Background informations:

http://www.vmware.com/de/business-critical-apps/exchange/microsoft-support.html
Please check this link and the whitepapers linked on the bottom. The whitepapers containing outstanding background informations for your exchange platform design.

There are also 2 interessting Webinars from VMware
http://vee.am/bpexchangeVMware
http://vee.am/bpexchangeHyperV

For VMware and Hyper-V
- Do not go higher than 2-to-1 in Virtual CPU to Physical Core ratio. Microsoft strongly recommend a ratio of 1-to-1 on the host where the Exchange Server runs.
- Use thick provisioning for the disks for better performance but use thin provisioning for backup optimization.
- Give the OS disk enough spare space and place it on a fast storage system as well (do not place it on a datastore with hundreds of other VM boot volumes.

For Hyper-V:
- Do not use Dynamic Memory
- Reserve 2 physical Cores for the Host OS
- If you perform on Host Backups reserve Ressources for this as well (RAM/CPU)
- Use VHDX whenever it is possible (max 64TB + improved sector alignment + more corruption resistent on power failures)
- Don´t forget to plan the space for the *.bin file (equal to memory size)

Check if you have the latest Exchange Updates installed. For Example:

Cumulative Update 3 for Exchange Server 2013

to address random backup problem with Event ID 2112 and 2180

4. More Veeam specific Exchange Tips and Tricks:
In general, if you use virtualization based backups for Exchange, there is a chance to hit one of 2 problems:

- Exchange DAG cluster failovers
- Exchange VSS timeouts (Exchange hard coded 20 second timeout between start of consistence porcessing and release.
Also if you design your Exchange environment, the next tips can help to prevent these problems as well.

Exchange DAG uses a network based heartbeat between the DAG members. When VMware Snapshots are committed, the delta changes are written back to the original storage space in small chunks. This is done by holding the VM (for a time the application can
not detect by default - storage latency) writing the chunk with part of the snapshot data back to the original storage space and release the VM. This will be repeated till all data is written. Aggressive DAG Cluster heartbeat can detect one or more of that VM holds as a network outtake and may perform an DAG cluster failover. This is transparent to the users in most cases. Multiple cluster failover/failbacks bringing extra load to the Exchange background services (e.g. indexer) and therefore you should prevent that situation.



To address this, check out the following tips and tricks:

Tips for preventing Exchange DAG cluster failovers:

Cause:

In general you need to prevent a big snapshot file by reducing disk block changes at backup time window and hold the snapshot lifetime at minimum possible, because less time means less changes in the snapshot file and finally less problems at snapshot commit.

Tips:

a)
Increase the DAG heartbeat time to avoid cluster failover (no reboot or service restart needed, they are online after you press enter).
On a command line (with admin rights)
cluster /prop SameSubnetDelay=2000:DWORD 
cluster /prop CrossSubnetDelay=4000:DWORD 
cluster /prop CrossSubnetThreshold=10:DWORD 
cluster /prop SameSubnetThreshold=10:DWORD

You can check the settings with:

cluster /prop

After setting this Registry Key, perform an manual Snapshot at VMware Client and release it after 3 seconds.

If a DAG failover is performed, please check tips below and work with VMware Support till this works without failover. After that you can work on the Veeam side by optimizing backup infrastructure.

b)

Use new Veeam Storage Snapshot Feature (StoreVirtual/3PAR/VSA) if you can (after v7 release) => Reduces Snapshot Lifetime to some seconds => No load and problems at commit because of less data. (This option can be counterproductive if you experience the 20 sec VSS timeout)

c) 
To reduce Snapshot commit time (and to reduce data in the snapshot), try to avoid any changes at the backup time window (User, Background processes, Antivirus, ....). Also try to avoid that on all LUNs on the storage System itself (faster writes at snapshot commit).
d)
If you cannot avoid many changes on block level at your backup window? Use Forward Incremental or if you need space Forward Incremental with daily transform into rollbacks. Reverse Incremental took a bit longer than the other backup methods => longer snapshot lifetime => more changes in the Snapshot to commit
e)
To reduce Snapshot lifetime and reduce amount of data to snapshot commit, use new Veeam parallel processing with enough resources to backup all of your disks at the same time (after v7 release)
f) 
To reduce backup time window and snapshot lifetime, use Direct SAN Mode with minimal needed disks connected at selected Proxy. If not possible use NBD mode with 10GbE. (Do not run Proxy in auto select mode). Disable VDDK Logging for Direct SAN Mode if your backups themselves run stable (ask support for the registry key and consequences).
g)
Use actual VMware Versions (newest VADP/VDDK Kits with a lot of updates in it) and actual Veeam Versions (newer VDDK Integration). And install actual ESXi/vCenter patches!
h) 
Use at minimum VMware vSphere 5.0 because of changes in the snapshot places and Background things.
i)
Still problems: Use faster disks for all of the VM disks (do not forget to place the OS disks on fast storage !!!)
j)
Less VM VMware disks can help to reduce snapshot commit time. Increase of maximum parallel snapshot commits setting can help to reduce the needed snapshot commit time (parallel vs. sequential). Check with Veeam support the registry settings and needed storage performance for this.
k)
In a worst case scenario and no other tips help, you can check the following VM setting. This an undocumented VM setting and you have to check with VMware the support statement. This was a tip from one of my customers with 13TB+ Exchange environment, who had a long run with VMware Support.
snapshot.maxConsolidateTime = "1" (in seconds) (again do this only together with VMware support). 
l)
If you have problems with cluster failover at Backup, one option is to backup DAG member(s) that hold only inactive databases (no cluster failover because of no active databases) (Logfile Truncation will be replicated by Exchange in whole DAG). This give you also the option to restart the server or services and Exchange process VSS consistency more faster afterwards. If you restart the services, take care that you wait long enough afterwards that also the VSS Exchange writers come up again, before you backup.

If you add an additional DAG member server for this, you have to check with you Exchange Architect the situation, because you change the member count for quorum failover selection. (e.g. you have 2 Exchange DAG members on different datacentres and a whiteness disk on datacentre 3 and you add an additional Exchange Server on one side, the failover is affected because of different server count on the one datacentre.)  


Tips for preventing VSS (timeout) problems:

Cause:
If you perform an VSS based consistency on an exchange server, hard coded 20 second timeout release the Exchange VSS writer state automatically if the consistency state are hold longer than these 20 seconds. The result is that you cannot perform consistent backups.
In detail, you have to perform Exchange VSS Writer consistency, VM snapshot and Exchange VSS writer release in these from Microsoft hard coded 20 seconds

Tips:

m)
If you see Exchange VSS Timeout EventLog 1296 => Change Log setting => 
Set-StorageGroup -Identity "<yourstoragegroup>" -CircularLoggingEnabled $false
n)
In many cases Exchange can perform consistency more faster if you add more CPU/Memory to the VM. Based on customer feedback this solved many of the VSS timeout problems.
o)
Use faster disks for all of the VM disks (do not forget to place the OS disks on fast storage as well !!!)
The worst thing you can do is to place the OS disk on a datastore with hundreds of other boot vmdk volumes on a Raid5/Raid6 storage.
p) 
Use at minimum VMware vSphere 5.0 because of changes in the snapshot creation area.
q)
Use actual VMware Versions (newest VADP/VDDK Kits with a lot of updates in it) and actual Veeam Versions (newer VDDK Integration). And install actual ESXi/vCenter patches!  => Perform Snapshots more faster
r)
Important one on VMware side: Less VM disks will reduce snapshot creation time. Check how long it take to start a snapshot of the VM in VMware vSphere (web) client. Think about that you need to perform Exchange VSS writer consistency + VM snapshot in 20 seconds.
s)
To optimize snapshot creation time: 
Check your vcenter load and optimize it (or use direct ESX(i) Connections for Veeam VM selection, so that the snapshot creation took less time.)
t)
Check your health an configuration of Exchange itself. I saw some installations where different problems ended up with a high cpu utilization at indexing service. This prevented VSS to work correct. Check also all other mail transport-cache settings. Sometimes the Transport Service cache replicate shadows of the mails over and over again and nobody commit them (if you have multiple transport services together with firewalls between them).
u)
Veeam specific: Veeam performs VSS processing over the network. Check with Veeam UserGuide TCP Port Matrix that B&R Server can perform Veeam Guest Processing over the network (open Firewall Ports).
If this is not possible Veeam failback (after a timeout) to networkless VMware Tools VIX communication channel (Veeam own) In-Guest processing.  If you use networkless In-Guest processing, change the veeam registry key, so that VIX based processing is performed before network In Guest processing. => No wasted time because of waiting for timeout.
However network based In Guest processing is performed faster and I  recommend it.



Additional Tips:

Use the Veeam Forums http://forums.veeam.com and search for specific Exchange Topics, there you can find additional tips and feedback. Keep in mind that Veeam Forum is not a official support forum. If you need urgent help, please open a support ticket http://www.veeam.com/support . Testing and Proof of concept environments have support with lower priority.

Do you have feedback?
Was one of the tips helpful?
Please send me feedback.

All the best to you and success... Andy

Montag, 24. Februar 2014

Exchange (DAG) VMware Backups: Updated list of tips and tricks for Veeam Backup & Replication

Please find updated Version here:
http://neufert-at-veeam.blogspot.de/2014/03/exchange-dag-vmware-backups-updated.html

Prioritisation of Veeam Backup & Replication Proxy Modes from my field experience.

Hi everybody,
just want to share with you a short list of Veeam Backup & Replication Proxy modes, because I got so many questions about it in the past.

Backup from VMware Block Storage:
From most recommended (up) to lowest priority (down)
Direct SAN Mode FibreChannel and Storage Snapshots (HP 3PAR StoreServe) on a physical Server
Direct SAN Mode iSCSI with Storage Snapshots (HP Lefthand StoreVirtual incl. VSA) on a physical Server
Direct SAN Mode iSCSI with Storage Snapshots (HP Lefthand StoreVirtual incl. VSA) on a VM
Direct SAN Mode FibreChannel/Shared SAS on a physical Server
Direct SAN 10GbE iSCSI on a physical Server
Direct SAN 10GbE iSCSI on a virtual Server
NBD (Network Mode) 10GbE (VMkernel Interfaces on 10GbE)
Direct SAN 1GbE iSCSI
HotAdd (Virtual Appliance Mode)
NBD (Network Mode) 1GbE (VMkernel Interfaces on 1GbE)

VMKernel Interfaces are limited to nearly ~40% of the interface speed for backup from VMware. So 10GbE is good while 1GbE Interfaces are too slow.
All Backup Modes including HotAdd work pretty stable with Backup & Replication , but VMware HotAdd is not so matured from VMware side then the other modes. So we try to use others backup modes in the field if possible. Workaround for some problems can be to roll out a Virtual Proxy in HotAdd Mode on each Host and set a registry key in Backup & Replication that force the backup job to use a proxy on same host as the VM that is processed.


Backup from NFS VMware Storage:
From most recommended (up) to lowest priority (down)
NBD (Network Mode) 10GbE (VMkernel Interfaces on 10GbE)
HotAdd (Virtual Appliance Mode)
NBD (Network Mode) 1GbE (VMkernel Interfaces on 1GbE)

VMKernel Interfaces are limited to nearly ~40% of the interface speed for backup from VMware. So 10GbE is good while 1GbE Interfaces are too slow.
All Backup Modes including HotAdd work pretty stable with Backup & Replication , but VMware HotAdd is not so matured from VMware side then the other modes. So we try to use others backup modes in the field if possible. Workaround for some problems can be to roll out a Virtual Proxy in HotAdd Mode on each Host and set a registry key in Backup & Replication that force the backup job to use a proxy on same host as the VM that is processed.


Veeam Backup & Replication Proxy Mode Autodetection process works like this:
From most recommended (up) to lowest priority (down)
Direct SAN Mode
HotAdd (Virtual Appliance Mode)
NBD (Netowork Mode)

So if you want to use 10GbE NBD Mode instead of HotAdd as default, you have to select it manually at the Veeam Backup & Replication - Backup Infrastructure - Proxy settings.



VM Restore (classic):
From most recommended (up) to lowest priority (down)
NBD (Network Mode) 10GbE (VMkernel Interfaces on 10GbE)
HotAdd (Virtual Appliance Mode)
NBD (Network Mode) 1GbE (VMkernel Interfaces on 1GbE)

By design HotAdd Restore produces much more I/O on Target Storage (~3x).
DirectSAN mode for restore is possible at VMware API but not used from Veeam for performance and stability (and therefore safety) reasons.

Instant VM Recovery by Veeam vPower NFS (datastore):
From most recommended (up) to lowest priority (down)
NBD (Network Mode) 10GbE (VMkernel Interfaces on 10GbE)
NBD (Network Mode) 1GbE (VMkernel Interfaces on 1GbE)



Mittwoch, 9. Oktober 2013

New Video (German) about Exchange Backup and VMware related challanges.

Hi everybody,

I did a new updated Exchange Backup Video for Veeam in German.
There are a lot of tips and tricks for general Exchange Backup at VMware, but also cover Veeam Exchange Backup and Restore.
Specific Exchange DAG settings are also discussed.

I hope to get a lot of feedback from you ;-)

http://www.veeam.com/de/videos/der-betrieb-von-exchange-auf-vmware-de-3187.html


CU Andy

Sonntag, 29. September 2013

Interessting numbers from IBM: 888PB per day data will be created every day ...

Hi,

got an email today from IBM - CH with 2 interessting numbers.
They say that each day 888PB will be created.
Also 90% of all data was produced in the last 2 years.

Dieses Blog durchsuchen

Wird geladen...