Question about 2.13.14

After updating to 2.13.14, DevKinsta is not working. Referring to Important DevKinsta update - 2.13.4, are the only options to wait for another release or delete the application & docker completely and start over?

Hi @madeatdawn! Thank you for reaching out to us, and welcome to the Kinsta community!

I am very sorry to hear that you have been affected by this issue. This is a unique issue that appears to only be affecting users who had either initially installed DevKinsta version 2.13.3 and then updated to 2.13.4, or had updated an existing DevKinsta installation to 2.13.3, and then again to 2.13.4. I certainly understand the frustration this type of issue can cause, and I assure you we want to help you get DevKinsta running again as soon as possible!

The currently available solution is to open Docker, and delete all containers, volumes and images related to DevKinsta. Upon relaunching DevKinsta the updated images will be pulled, the new containers created and the volumes will be recreated. DevKinsta should then be able to launch. This will however result in the loss of all local siteā€™s databases. This means all post/page content, and site configuration of all local sites that exist under your DevKinsta installation will be permanently deleted.

Our development team is looking into additional options to see if there is a way to resolve the issue while preserving data.

One thing you can try however is to delete all containers and images related to DevKinsta within Docker, but preserve the volumes. Then, try relaunching DevKinsta to see if that helps allow it to load. The database information should be preserved as that is stored within the devkinsta_db_dat volume.

Please let us know if this helps! We will also provide another update as soon as possible once we have more details to share regarding this matter.

We appreciate your patience while our team continues to look into solutions.

Hi Andrew,
Iā€™m pretty much stuck in the same situation here, with DevKinsta unusable and I client sites staged locally, that I need to push to the hosting environment.

I understand the difficulty with coding and development and all - but Iā€™m stuck here and I need to make some decisions very fast. Can you give us an indication that:

  • devs will likely/not likely have a fix for 21.3.4
  • Das will like/not likely be savable.
  • fix is coming in hours/days/weeks.

That would help a bunch of us who might need plan around this issue. I appreciate the understanding!

1 Like

@klagreca2

I understand your concern. Our DevKinsta developers are working to find a solution, but at this time, we donā€™t have confirmation of whether there will be a better solution beyond what Andrew has already suggested. We know that itā€™s frustrating not to have a clear solution and timeframe. Please know that we will continue to provide updates as soon as we hear back from our DevKinsta developers. Thank you for your patience.

Why was this update not pulled? Now I have to spend all this time to reconfigure everything because you pushed a bad update. Thanks a lot.

Iā€™m turning off auto-update so this doesnā€™t happen again.

@halshak has a good point. If you folks havenā€™t pulled this update, that should be the first thing you should do.

1 Like

DISCLAIMER: This may destroy your data, I am not a MariaDB expert and this solution is specific to this Devkinsta issue and MariaDB 10.5.17 (version <10.8). Deleting the file ib_logfile0 is not a trivial action.

Caveat: I did not update my sites or data at all after this Devkinsta issue presented itself. Iā€™m not sure the consequences of doing the following if youā€™ve been restoring/importing backups. I am on macOS Sonoma.

Here is what I did to get DevKinsta working and be able to get my data out:

  • Stop all running sites and exit DevKinsta
  • Open Docker Desktop and make sure containers are stopped
  • In Docker Desktop go into Volumes and create a clone of devkinsta_db_data THIS IS IMPORTANT OR YOU MAY EVENTUALLY LOSE ALL YOUR DATA
  • AGAIN: You should clone that volume or you could be SOL and lose everything, as the Devkinsta folks have repeatedly mentioned here, that volume is where your databases are stored.
  • Optional: I would go into the cloned volume, should be named something like devkinsta_db_data_cloned and export to your local disk (this will create a .tar.gz export file)
  • Go back into the devkinsta_db_data volume and delete ib_logfile0
  • Start DevKinsta back up

This worked for me, at least to get sites running, be able to export db data, etc. The file I deleted, ib_logfile0, is not a trivial file. It is the ā€œredo logā€ and is essential for database function, but from extensive googling: occasionally that file when deleted (and then recreated) can solve some db issues. Also, since the DevKinsta MariaDB version is <10.8 deleting ib_logfile0 doesnā€™t obliterate the database. In newer versions of MariaDB deleting this file can result in corruption of your data.

Again, YMMV. Also, since I had already cloned the volume and was playing around with the devkinsta_db_data volume trying to get this fixed, what I actually did was import the devkinsta_db_data_clone backup into the devkinsta_db_data volume and delete the ib_logfile0 file, but I donā€™t think the extra steps of emptying the volume and importing the clone backup is necessary (youā€™d just be putting back the data you cloned).

Hopefully this makes sense and hopefully the DevKinsta folks will be able to solve this for everyone without risky workarounds or wiping all the data. :slight_smile:

1 Like

Thank you so much for your advice. Iā€™m posting here just to say thank you, BocaJuniors.

Followed your advice and now my devkinsta_db restart loop is stopped.
(still working on to recover - now the local website says ā€œError establishing a database connectionā€)

Iā€™ve got a problem with my devkinsta after the automatic update of 2.13.3 ā†’ 2.13.4
and after the update, I realized that I did the wrong thing. Important DevKinsta update - 2.13.4

Right after the update to 2.13.4 (yes, I did double click devkinsta.exeā€¦) , my devkinsta_db container got stuck in restart loop and cannot access the local sites on my devkinsta.

Restart docker, Restart devkinsta - no effect
Logs - it says only mysqld (mysqld 10.5.17-MariaDB-1:10.5.17+maria~ubu2004) starting as process 1 ā€¦
Docker shows devkinsta_db start - stopped - start repeat.

But after the emergency fix, restart loop is stopped, and now Iā€™m seeking the solution for the next problem ā€œError establishing a database connectionā€.

While struggling, it seems Iā€™ve fixed the problem.
(please note that this maybe not a right solution. Please wait for the devkinsta official fix annoucement)

On Windows PowerShell (Iā€™m on WSL2, windows 10 pro)
Command:
docker network connect devkinsta_network devkinsta_db

Then, devkinsta_db will be connected to devkinsta_network, and now my local sites all seem to be working.

Would it be possible to simply uninstall 21.3.4 and reinstall 21.3.3, or will uninstalling wipe all my containers from Docker?

Also would need to locate a 21.3.3 specific installation file

That was what Iā€™ve tried to do.
I didnā€™t have 2.13.3 files, so I had to find another way.

Can you elaborate more on specifically what youā€™ve done with a full procedure?

Iā€™ve attempted Boca Juniors fix and while it does resolve the issue of the constant restarting of the container, but as per @dekindock, it seems that the root password for MySQL gets changed as the websites still canā€™t connect and attempting to login directly via the DB manager shows ā€œconnection refusedā€.

Has anyone managed to actually get things fully up and running, or even just able to access AdMiner so DBs can be exported?

Hello @Adam_Hodson

Did you try the solution from Andrewā€™s reply?

One thing you can try however is to delete all containers and images related to DevKinsta within Docker, but preserve the volumes. Then, try relaunching DevKinsta to see if that helps allow it to load. The database information should be preserved as that is stored within the devkinsta_db_dat volume.

Please let me know if you tried that.

I have tried everything in this thread and while one of the solutions results in the constant restarting issue being resolved, Iā€™m still unable to connect to the MySQL server afterwards.

In the meantime, Iā€™ve worked out a solution where Iā€™ve been able to convert the .frm and .ibd files stored on the devkinsta_db_data volume back in to MySQL while I await a fix.

Iā€™m going to be sharing a write-up on how that is done here later today; Iā€™m hoping that you guys are able to roll out a fix, though, as this is a significant issue.

2 Likes

Hello @Adam_Hodson :wave:

Thank you, posting your workaround might help others so we truly appreciate it.

So far, our devs are still looking into this. Apart from the solution that would cause a loss of data, we donā€™t have a fix yet, but weā€™ll keep you posted.

Hi Adam, I appreciate the effort and let us know where we can send you flowers and/or beer! I would hope though, that someone from the support team would also be documenting a manual fix. @VladimirM can we make that happen? Itā€™s going on a week now and many of us have client work backing up.

Hi Everyone -

Iā€™ve identified a few ways that you may be able to restore your website data, both of which will involve using the .frm and .ibd files found within your devkinsta_db_data volume.

1. First you will need to export the data from your volume.

  • Navigate to the volume within Docker Desktop and click into the ā€œStored Dataā€ tab.
  • You will see directories for each of your databases that will have a series of .frm and .ibd files within them that match your table names.
  • Right click on the directory and click ā€œSave Asā€ and save it to your local disk.
  • Next find the file ā€˜ibdata1ā€™ in the root of the ā€œStored Dataā€ directory and save this inside of the same location you just saved the previous files.

Now you will have three potential ways to restore your actual database. You will need access to PhpMyAdmin or some other MySQL server/interface to use the below methods.

  1. Using dbsake to restore from .frm and .ibd files (most complex but free method that can be done without Python)
  • Reference this github for specific instructions: Recover MAMP MySQL database using .frm and .ibd files after InnoDB crash Ā· GitHub
  • Instead of running ./dbsake frmdump /path/to/database/folder/filename.frm for each table, you can run ./dbsake frmdump --recursive /path/to/database/folder/
  • This will generate SQL statements that will allow you to recreate all of your tables within PhpMyAdmin
  • Navigate to PhpMyAdmin, create a new database, click the SQL tab, and paste the generated queries. This will create the tables.
  • Now follow the remaining instructions within the Git to alter the tables and remove / replace the tablespace with your .ibd files.
  • NOTE: I had limited success with this option. I believe itā€™s due to a mismatched ibdata1 file and/or mismatched ib_logfile file. I did not want to potentially break my MAMP setup as well so did not attempt to replace these files with the ones from Docker, but if you are working off a fresh MAMP install with no other DBs it may be worth a try.
  1. Database Restore .Frm Python 3 using mysqlfrm (still complex method but free)
  1. Use Aryson Mysql Database Repair
  • Before anyone asks, I have no affiliation whatsoever with this company, I found them through trial and error on Google.
  • You will need a PC to use this method
  • Download the demo program to make sure your files will work: MySQL Database Recovery Tool to Repair MySQL Database Tables
  • Once you download the demo open it up and click the ā€œOpenā€ button.
  • Choose the folder that contains your .ibd files, .frm files, and ibdata1 file. All of the files must be in the same directory (the ibdata1 should not be outside of the folder that contains the other files).
  • It will then read all of the data and recreate the DB tables in its GUI. You should be able to click into each table and preview your data. It will only let you preview though, but wonā€™t let you save using the demo version.
  • Once you can confirm it will work, you can buy the software here: https://www.arysontechnologies.com/mysql-repair/buy-now.html
  • When you save the MySQL file, you will have the option to save directly to a database (this didnā€™t work for me as I wasnā€™t sure how to configure the PC to work even though my connection details worked. PC people may have more luck with this.) or you can save the script output (this is what I did).
  • Once the script output is saved, you will see a new folder that will have a SQL file as well as a folder with blobdata.
  • If you open up the SQL file in a file editor you will see how the blob data is being used, i.e. INSERT INTO wp_options(option_id,option_name,option_value,autoload) VALUES(30,ā€˜rewrite_rulesā€™,LOAD_FILE(ā€˜/path/to/blobdata/1ā€™),ā€˜onā€™);
  • Depending on your setup and where youā€™re creating this new DB, you may be able to use LOAD_FILE, however, I found it easier to just replace the LOAD_FILE() calls with the actual data from the blob files (you can open these in a text editor and youā€™ll see serialized data).
  • If you have to replace a lot of blob files in the query, you may want to write some PHP to just read the blob files and generate the SQL query string for you, or you can just copy and paste whichever is easier.

If all of the steps in number 3 are followed you should then be able to just execute the SQL within PhpMyAdmin.

I know this is not the cleanest solution, but it will work to at least get your DBs back so you can then re-import into DevKinsta or another local dev environment if you donā€™t plan to continue using DevKinsta moving forward.

@klagreca2 @BocaJuniors @Mark @dekindock @halshak @madeatdawn Not sure if my other reply will notify you but posted above are a few options that can work.

Maybe Kinsta will credit the $149 cost for the software solution on a future hosting invoice :wink:

@Adam_Hodson Youā€™re braver than I wasā€¦after initially going down the .frm rabbit hole, I opted for something less technical. Kudos, impressive.

Interesting that you both had an issue with the root password being changed, I wonder if deleting the devkinsta_db container (not the volume) and allowing for recreation by DevKinsta on launch would resolve? Odd that I never ran into that.