WhatsApp 
Veeam Backup & Replication has gradually established itself as one of the most popular solutions for many companies, ensuring the backup and recovery of their virtual, physical and cloud environments.
Unfortunately, even the most robust systems can fail…
One of the most critical error messages that Veeam users sometimes encounter is: “All instances of the storage metadata are corrupted”.
How should this error be interpreted, and why is it so serious?
Veeam storage metadata works roughly like a backup GPS. Without it, Veeam is at a total loss, unable to locate or correctly assemble backed-up data.
What makes this error particularly alarming is that Veeam normally creates two copies of this metadata precisely to avoid this type of situation.
When the message reads “All instances are corrupted”, this means that both copies are unusable. It‘ s as if your Plan A and Plan B had failed simultaneously.
The help of a specialist to quickly solve this Veeam backup problem will be of great assistance.
What are the potential causes of this metadata corruption?
There are several possible causes of this error…
Problems with backup storage are a common cause. This can include hardware malfunctions (faulty hard disks, failed RAID controllers), storage firmware problems, or communication concerns, particularly when using network shares via SMB on low-end NAS devices which may not handle synchronization commands (FLUSH) correctly.
Unexpected interruptions during the backup process, such as a reboot of the Veeam server, can also corrupt backup files.
In more recent cases, problems linked to specific Veeam versions or interactions with file systems such as ReFS approaching their maximum capacity have been reported.
Finally, a ransomware infection targeting backup files can also cause damage to metadata, which is usually particularly targeted in an attack… Many medium-sized companies find themselves faced with the same error after an attack that has encrypted their VBK files.
Our emergency hotline is available 24/7/365 to help you in such situations.
Facing the emergency: the first steps to take
If you see this error during a restoration attempt, don’t panic. Your first action should be totry a different restore point. It’s not uncommon for only part of a backup to be affected.
If the error appears during a backup job, here are our recommendations, based on real-life situations:
- Immediately create a new backup chain. This is the official Veeam solution, and usually works well to isolate the problem.
- Examine your storage. Run a CHKDSK (Windows) or fsck (Linux) to identify any underlying problems. This simple diagnostic may reveal faulty sectors on a non-suspicious disk.
- Dive into the event logs. They often contain valuable clues to what’s really going on. Thorough troubleshooting can reveal repeated I/O errors occurring before each metadata corruption.
- Test your storage by manually copying large files. This can help reproduce and identify the problem. This technique can sometimes confirm that a faulty RAID controller is the real culprit.
Don’t overlook Veeam’s built-in tools, such as SureBackup or Veeam.Backup.Validator.exe. They’re there for a reason. A full scan can reveal problems before they become critical.
When all seems lost: last resort solutions
In the most serious situations, especially after a ransomware attack, special measures are called for. Recoveo has developed specialized techniques for recovering data from corrupted Veeam backups.
Our experts successfully extract entire virtual machines and critical databases from VBK files that Veeam itself considered unrecoverable. These interventions can make the difference between a rapid recovery and a prolonged disaster.
Constant technological innovation
Since 2019, we’ve been developing powerful tools to optimize Veeam backup recovery. Our first tool, Backup Recovery, enabled us to restore the integrity of VBK files and make them usable.
In 2023, we developed Backup Extractor, a more advanced solution designed to respond to new threats and the complexity of ransomware attacks. This tool is now capable of in-depth analysis of the structure of backup files, reconstructing tree structures and restarting virtual machines under Microsoft Hyper-V and VMware ESXi, for example.
An approach adapted to new challenges
With ransomware on the rise, we need to keep innovating. That’s why our experts work relentlessly to improve our tools and guarantee effective, rapid data recovery. We cover a wide range of Veeam versions, from 6.5 to 12.1.1, enabling us to intervene on a variety of infrastructures. What’s more, our Diagview tool provides post-recovery integrity checks, offering total transparency on restored files.
In-lab or remote intervention
We offer two types of recovery: in-lab and remote.
Laboratory intervention is ideal for the most complex cases, particularly where data has been permanently deleted.
Remote recovery, on the other hand, enables rapid, efficient intervention without immobilizing infrastructure, thus reducing logistical costs and ensuring continuity of service. Our high-speed connection and use of secure protocols such as SFTP guarantee data protection and integrity during transfers.
Thanks to our expertise and commitment to innovation, we offer a reliable, high-performance solution for Veeam backup recovery, enabling businesses to overcome cyber-attacks and recover all their data safely.
Prevention rather than cure: proactive strategies before disaster strikes
The best way to deal with such an error is, of course, never to encounter it… Here are the practices we systematically recommend:
The 3-2-1 rule is non-negotiable. Three copies of your data, on two different media, including one off-site. This strategy has literally saved many companies.
Invest in quality storage. A professional server or a properly configured SAN system will considerably reduce your risks. The difference in reliability is remarkable and justifies the investment.
Schedule regular integrity checks. Corruption detected early can often be isolated before contaminating other backups. Preventive audits can identify incipient problems that could turn catastrophic a few weeks later.
Maintain active monitoring of your backup infrastructure. Warning signs such as slowdowns or minor errors can herald more serious problems to come.
A realistic approach to the inevitable
Years of experience with Veeam and the resolution of these critical situations show that a proactive approach, combined with a well-defined reaction plan, makes all the difference.
The “All instances of the storage metadata are corrupted” error may be alarming, but it’s not a death sentence for your data. With the right knowledge and resources, this potential crisis can be turned into a setback. We are at your disposal to help you solve this type of problem, so don’t hesitate to get in touch.
FAQ
There are several possible causes of this error. Problems with backup storage are a common cause. This can include hardware malfunctions (faulty hard disks, failed RAID controllers), storage firmware problems, or communication concerns, particularly when using network shares via SMB on low-end NAS devices which may not handle synchronization commands (FLUSH) correctly. Unexpected interruptions during the backup process, such as a reboot of the Veeam server, can also corrupt backup files. In more recent cases, problems with specific Veeam versions or interactions with file systems such as ReFS approaching their maximum capacity have been reported. A ransomware infection targeting backup files can also cause metadata damage.
This suggests that even if the initial full backup was written without any apparent data-level errors, the metadata recorded at that time was already corrupt or subsequently became corrupt. This may indicate an underlying problem with the backed-up system itself, or an intermittent problem with the destination storage that affects the integrity of the written metadata.
The first step suggested by the Veeam teams is often to try a more recent BETA version of the software, as patches may have been made. If the problem persists, it is advisable to provide detailed information on the configuration of the affected machine (disk configuration with lsblk -af, output of dmesg -T, presence of RAID, backup mode, backup target and Veeam logs located in /var/log/veeam). Testing a different backup target (local USB disk, NFS share instead of CIFS) can help determine whether the problem is related to the destination storage. If other machines with a similar configuration back up to the same target without any problems, the problem is more likely to be specific to the affected machine.
Similar to the Linux version, it is advisable to open a support ticket for in-depth log analysis, especially with the free version where direct support is limited. However, the cause is often storage corruption. Even if USB hard drives are replaced, the problem may lie in the NAS to which they are connected via SMB. The use of SMB to a low-end NAS has been identified as a frequent source of storage corruption. The main recommendation is to perform new backups, preferably to new, reliable backup storage.
Officially, Veeam does not provide standard tools for manually repairing the header or metadata of corrupt VBK or VIB files. In the event of corruption due to ransomware, it is advisable to contact Veeam support, although there is no guarantee of recovery. Third-party companies specializing in data recovery from ransomware attacks, such as Recoveo, may offer specific tools and techniques for attempting to extract data from corrupted or encrypted Veeam files. They point out that recovery attempts with generic software can often result in corrupted files.
Veeam uses metadata files (such as VBM files associated with VBK and VIB files) to keep track of information on restore points, backup configuration, backed-up objects and the location of data blocks in backup files. For redundancy, two identical instances of storage metadata are usually included in backup files and are never updated simultaneously. This approach is designed to ensure that at least one copy remains valid in the event of a crash or corruption during updating. Corruption of both instances makes it extremely difficult to understand the structure and content of the backup files, preventing normal restore operations.
Several measures can reduce the risk of corruption. Using reliable, high-performance backup storage capable of correctly handling write operations and synchronization commands is essential. Avoiding sudden interruptions to backup processes is crucial, which means ensuring a stable power supply for backup servers and storage. It’s also advisable to follow the 3-2-1 rule for backup: have at least three copies of your data, on at least two different media, including one off-site copy. Regularly test restores using features such as SureBackup to verify the integrity of backups and metadata. If you use NAS, make sure they are correctly configured and kept up to date.
