There are plenty of misconceptions while moving the workload from Physical to Virtual world. Customer sometimes assume that while virtualizing their infrastructure concepts application behavior may change – which is not the real case. Recently I was working on one engagement with one of VMware’s large customer and they are planning to virtualize few core application from physical to virtual environment. While in planning phase some questions came across which dragged my attention to have a ground zero level discussion with the customer and the key thing which came out of that discussion was – no matter if the application is running on Physical or Virtual infrastructure we should give equal importance to the thumb rules of assigning vCPU/Memory and Storage to these application.
In this blog I’ll be covering majorly about 3 large enterprise applications Enterprise JAVA, Oracle & SQL.
Enterprise JAVA: Any JAVA application will have primarily 4 components which needs to be tuned perfectly no matter the application is running in physical or virtual world
Load Balancer: which will provide load balancing algorithms & API integrations with vSphere. This tier is basically responsible for handling application traffic demands
Web Server: this tier should be tuned according to the exact number of HTTP threads to cater application demand
JAVA Application: many JAVA based applications gives the ability to tune JVM to cater the traffic demand via Java threads, JDBC or GC parameter. The same rule applies on it while you are virtualizing your application and similar tuning parameter needs to be in set here as well
DB Server: Databse is the heart of any application and VMware has time and again proved significant for running various databases on VMware platform and similar best practice examples can be considered here as well
vCPU Best Practices JAVA Workload: There are some vCPU best practices which needs to be considered while running JAVA workload on virtual environment:
- Create a workload profile and conduct a load test to measure as how many JVMs can be run on a particular sized virtual machine. Using this test create a best case scenario as how many concurrent transactions can be pushed to a configuration before it can be called for scaling horizontally in a cluster
- For enterprise application which are considered business-critical make sure for those vms at any point total number of vCPU assignment doesn’t cause greater than 80% of CPU utilization to ESXi host
- Oversizing your vCPU is equally bad as under-sizing it. So if performance load test results says 2 vCPU is adequate to run any application but you go ahead and assign 4 vCPU to that virtual machine then 2 vCPUs will be sitting idle at most of the time which isn’t consider optimal.
Memory Best Practices for JAVA Workload: Understanding memory is very important for any application and if it is JAVA then it becomes even important because JAVA uses memory differently than other applications.
Here is the image to understand various memory segments of JAVA thread here:
VM Memory = Guest OS Memory + JVM Memory
JVM Memory = JVM Max Heap (-Xmx value) + JVM Perm Gen (-XX:MaxPermSize)
+ NumberOfConcurrentThreads * (-Xss)
- -Xms is the value determined during load testing and it remains no change whether you are running your workload on Physical or Virtual environment
- It is recommended not to over commit memory because JVM memory is an active memory where objects are constantly being created and garbage collected. Such an environment will always require Active memory and over committing memory may result into Ballooning or Swapping to occur which may degrade the overall performance
- JVM running on virtual environment will always require Active Heap to be present in physical memory so Reservation can be set for vm running JAVA workload Reservation Memory = VM Memory = guest OS Memory + JVM Memory
- JVM Perm Gen (-XX:MaxPermSize) is area in addition to -Xmx and it holds class level information of the code
If you have multiple JVM running on a VM then VM memory = guest OS memory + N * JVM
vSphere Best Practices for JAVA:
- Enable Hot add CPU and Hot add Memory on vm so that in the course of need this feature can be leveraged. This can be useful for vm to handle heavy traffic
- While increasing the Heap Size space typically one must increase vCPU count as well to get best GC cycle performance. Refer to respective JVM documentation for more details
- While running JAVA workload on vSphere platform various inhouse VMware concepts like HA, DRS and Affinity rules can be used in order to leverage minimal downtime, intelligent placement and utilizing better licensing respectively
- Enable Turbo mode is processor supports it and make sure hyper-threading is enabled in BIOS
- While on ESXTOP below metrics Threshold can be looked upon for any performance issue
Oracle: There are plenty of use cases around the globe where customer are successfully running Oracle on VMware environment and Oracle also in its official statement fully supports virtualization of Oracle database on vSphere platform and VMware also has total ownership documentation for the same. Which means that customer can log case with VMware and VMware support team will take up that case to Oracle and will work with Oracle team until the issue is resolved. Here is a glimpse of VMware support model for Oracle databse.
So here are some best practices to be followed while running Oracle database on vSphere platform:
vCPU Best Practices:
- Do not over allocate the number of vCPU and try to match it with exact workload
- A thorough understanding of your underlying workload is really important for right-sizing hence if the underlying application is commercial then follow published guidelines but if the application is custom-written consult with application developer to understand resource requirements
- If still unsure of resource requirement then go ahead and follow hardware vendor recommended Oracle sizing guidelines
- Use VMware vSphere Virtual Symmetric Multiprocessing (Virtual SMP). It enhances virtual machine performance by enabling a single virtual machine to use multiple physical processors simultaneously. vSphere 6.0 supports up to 64 virtual CPUs per virtual machine
- For Tier-1 Production Critical workload vCPU over-commitment isn’t recommended and the value should be 1:1 (1 Physical Core : 1 vCPU)
- Hyperthreading is seldom considered as a bonus to double the total processor core value, however in real world it merely provides 10-20% of performance improvement and it should be used very wisely
- Enabling Oracle NUMA support might improve performance but Oracle documentation suggests it to be tested in a test environment before deciding to use it with production system
Memory Best Practices:
- Set Memory Reservations of VM running Oracle workload equals to Sum total of the size of the Oracle SGA (System Global Area) , the Oracle PGA (Program Global Area) , the Oracle Background processes stack space and Operating System Used Memory (Note: Setting reservations can limit vSphere vMotion operations. A virtual machine can be live migrated only if the target ESX / ESXi host has free physical memory equal to or greater than the size of the reservation)
- ESXi (3.5 onwards) as well as Oracle (9i R2 for Linux & 10g R2 for Windows onwards) supports Large page in guest OS to improve performance improvements
- However Transparent Page Sharing (TPS) feature of vSphere claims DBAs to over-commit memory while minimizing performance improvements but this feature to be used very carefully in Production environment
- If Oracle database is part of a third-party commercial enterprise application (ERP), follow virtualization guidelines from the ERP vendor
Storage & Networking Best Practices for Oracle:
- For storage array that is not VMware vSphere Storage APIs – Array Integration aware format VMDK as eager zeroed thick. However for storage arrays with vSphere Storage APIs – Array Integration aware VMDKs can be thin provisioned
- Customer can choose between VMFS, RDM (physical), RDM (virtual) to deploy Oracle workload as the performance results are almost identical in all use cases. For some specific scenarios where RDM is the absolutely required customer are free to make a choice however VMFS being VMware native file system has some advantages while Oracle workload is running on VMware platform
- It is a best practice to carve dedicated datastore for I/O intensive Oracle workload. It will allow DBAs to define individual service level guarantees for different applications
- Use VMXNET paravirtualized network adapter because of its minimal overhead traffic pass through between vm and Physical network interface cards
- Use nic teaming for load balancing and better availability
- Enable Jumbo Frames for Oracle interconnect traffic on ESXi as well as physical switch level
- Use SIOC & NIOC for better QOS and leverage 10 GbpE network for storage traffic to improve performance
- However customer can choose from using Standard Switch (vSwitch) or Distributed vSwitch (vDS) but there are several advantages of using a vDS over vSwitch like centralized management,traffic shaping, LLDP, Netflow etc which can be considered
- A virtual machine housing an Oracle database has two types of VMDKs—guest OS VMDK and the VMDKs housing the Oracle data files. VMware does not recommend that you back up a high transactional, heavy I/O-centric Oracle database using VMware snapshot technology because, during the snapshot removal (consolidation), there is a brief stun moment. No activity is permitted against the virtual machine, which might result in performance issue and service disruptions
Microsoft SQL: is one of the most widely used databases around the world and there are plenty of customer who are running SQL server successfully on vSphere environment. VMware also has its own case studies for running SQL server and VMware fully supports MSCS. Virtualizing Microsoft SQL Server with vSphere enables many additional benefits. Using vMotion a running workload can be migrated from one physical server to another without downtime, DRS (Distributed Resource Scheduler) can help with intelligent placement of workload virtual machine to optimally utilize underlying ESXi host resources. HA (High Availability) or FT (Fault Tolerance) are another features which comes along with VMware provide cost effective high availability solutions.
Majority of vCPU/Memory best practices for deploying SQL server on vSphere will remain same as Oracle or any other database, however I would like to discuss about few of them.
Starting with NUMA considerations which will come into picture for critical virtual machines.
Example: VM with NUMA Locality: Let’s consider there is a VM with 6 vCPUs and 32 GB of RAM residing in a server that has 8 cores and 96 GB of RAM in each NUMA node, with a total of 16 CPU cores and 192 GB RAM. The ESXi server places the VM entirely in one NUMA node, making sure all processes stay local and performance is optimized
Example: VM with vNUMA: For wide SQL Server virtual machines, where the number of allocated vCPUs is greater than the number of cores in the NUMA node, ESXi divides the CPU and memory of the VM into two or more virtual NUMA nodes and places each vNUMA on a different physical NUMA node. There is a single VM with 12 vCPUs and 128 GB of RAM residing on the same physical server that has 8 cores and 96 GB of RAM in each NUMA node, with a total of 16 CPU cores and 192 GB RAM. The VM will be created as a wide VM with a vNUMA architecture that is exposed to the underling guest server OS.
*By default vNUMA is enables only for a VM with 9 or more vCPUs. For VM with more vCPUs than Physical Cores on physical NUMA, always assign a number of vCPU that can be divided evenly between physical NUMA nodes. This reccomendation doesn’t apply on VM with vCPU less than physical cores
Memory Right Sizing and Overhead: There are few best practices while assigning memory on VM running SQL server
- In order to avoid memory contention situation avoid memory over-commitment at host level (HostMem >= Sum of VMs memory – Overhead)
- Since SQL server is NUMA aware application hence wherever possible ESXi will avoid remote memory access as much possible
- Any VM will require a certain amount of overhead memory to power on, so please consider this value from below table
|Memory (MB)||1 vCPU||2 vCPU||4 vCPU||8 vCPU|
4. Consider Memory Reservation for critical workload
VM Memory = SQL Max Server Memory + ThreadStack + OS Mem + VM Overhead
ThreadStack = SQL Max Worker Threads * ThreadStackSize
ThreadStackSize = 1MB on x86
= 2MB on x64
= 4MB on IA64
OS Mem: 1GB for every 4 CPU Cores
RDM vs VMFS Volume: VMware supports Raw Device Mapping (RDM) which allows a virtual machine to directly access a volume from physical storage without formatting it in VMFS. RDM can also reach upto 64 TB. Please refer Table 3.3 of configurations maximum for detailed information. From a performance perspective both RDM as well as VMFS can provide similar performance characteristics and a detailed study on this has been documented in the link.
I have listed down some useful links which contains best practice information while running enterprise applications on vSphere platform. With this I end this blog.. I hope it helps.. till then Happy Reading 🙂
- Enterprise JAVA on vSphere
- Virtualizing Enterprise JAVA
- Virtualizing Oracle with VMware
- Oracle best Practices for VMware
- Understanding Oracle Support and Licensing
- Running SQL on VMware
- Microsoft SQL Server on vSphere
- SQL best practices for VMware