Toyota Car Assembly Lines Grind To A Halt After Running Out Of Hard Drive Storage
In a recent blog post, Toyota reported that regular maintenance work was scheduled and performed on August 27th, wherein data was being deleted or organized within a database on what sounds like a system that handles parts orders. This maintenance, however, led to an error due to insufficient disk space, causing the system to come to a halt. This was not the only system on the server, though, and as such, failures in backups and failover systems likely virtualized on the main server ultimately led to the shutdown.
Subsequently, 12 production plants and 25 production lines were shut down in Japan until August 30th, when they were restored. The problem was ultimately fixed when the database was moved to a server with a larger capacity on the 29th. Toyota has also stated that the company has “identified the above as the true cause” and that “Countermeasures have also been put in place by replicating and verifying the situation.”
At the end of the day, this situation highlights not only the intricacies of IT work but also the value of performing ongoing testing of systems and business continuity plans alike. It seems this was a lesson learned, as Toyota will review maintenance procedures and strengthen efforts to prevent this from happening again to “deliver as many vehicles to [its] customers as soon as possible.” Hopefully, this also serves as a lesson to other IT workers, as that which seems routine could break everything without even realizing it.
