Company has given an explanation of why it suffered an almost two-day outage

LucidLink logo(1)

LucidLink has contacted the FBI as it looks to investigate a cyber attack that left its file sharing service out of action for almost two days.

The comany has posted an explanation of what happened to cause the issue, which saw its service become unavailable from 13:55 on 29 April until 13:06 on 1 May. Broadcast Tech reported on the situation as it happened, here.

According to LucidLink, it was not a ransomware attack, and the company, “has no indication that any unauthorized access to personal, corporate, or other data was leaked or otherwise affected.” It believes its zero-knowledge encryption model, which means only those who have an encryption key can access files, is to thank for this.

It believes that, “the root cause of the event to be malicious exploitation of an internal server with access to the production environment. This server was utilized to gain elevated privilege and to execute a script that corrupted the disk attached to each metadata server.” 

LucidLink explains that most of its servers, totalling roughly 5,000 filespaces, became unresponsive at 13:55 on 29 April. When LucidLink clients become disconnected, they contact the discovery service to determine the IP address of the metadata server they need to reconnect to. Having all of the company’s clients attempt to do this at once overloaded the discovery service, leading the company to initially suspect a DDoS attack.

However, it then worked out that the conncetions were genuine, and that the disk attached to each of the servers was corrupted - and had been at the same time in multiple time zones on different continents. LucidLink examined the corrupted disks and found that large portions of the disks had been overwritten with random data.

Now aware of the issue, the company set about deploying many thousands of new metadata servers with the appropriate build image - which needed a new process to do so at such scale. It then recovered the metadata to fill these servers from its rolling six-hour backups, which again needed a new process to complete at a far larger scale and shorter timeframe than is usually necessary.

When this process was completed, 98% of filespaces were restored, with some needing manual intervention.

When the attack was discovered, LucidLink blocked all internet-facing traffic, restricted infrastructure access only to essential personnel, and ran extensive vulnerability tests and malware scanning. It then quarantined all servers with elevated privileges to access the production environment and continued with extensive analysis of network, systems, EDR, VPN, access, and application logs.

After credential rotations, replacing all API tokens, SSH key pairs, and local account passwords, LucidLink then contacted law enforcement such as the FBI’s Internet Crime Complaint Center, as well as third party specialists in cyber forensics and legalities. It has also updated its network topology and implemented additional access restrictions with the aim to further protect its server infrastructure.

In addition, LucidLink is dedicating additional engineering resources to security-related projects, implementing a security operations center for enhanced threat monitoring.

Incident Timeline

29 April, 2024
13:55 BST Approximately 5,000 filespaces became unresponsive.
14:05 BST The support team received the first ticket regarding the incident.
18:00 BST LucidLink notified customers that many filespaces were unresponsive.
18:16 BST LucidLink discovered that some metadata servers had corrupted disks.

30 April, 2024
14:13 BST LucidLink developed a program to automatically recover metadata servers from backups.
16:00 BST LucidLink scaled the discovery service horizontally to run on multiple instances in order to accommodate the surge in traffic.
19:00 BST LucidLink completed the provisioning of new metadata servers.
20:30 BST The first affected filespace was restored.

1 May, 2024
01:10 BST LucidLink started the automated restoration of filespaces.
03:55 BST The automated process restored 98% of affected filespaces.
11:00 BST LucidLink started manual restoration of the remaining 2% of affected filespaces.
13:06 BST All affected filespaces were back online.