I am running antiX 15 from a live usb with root persistence. When I place a load on the machine it seems to get into a state where the cpu is heavily used by the system. When in this state the monitor shows about 86% cpu usage with disk usage ranging 111M to 162M (approx). I ran top and found it showed the following:

Tasks 188 1 running 187 sleeping 0 stopped 0 zombie
%CPU .8 us 84.2 sy 0 ni 0 id 13.7 wa 0 hi 1.3 si 0 st
KiB Mem 51068 total 503492 wuse 6576 free 2700 buffers
KiB Swap 0 total 0 used 0 free 307932 cached Memory

Any ideas on what may be causing this condition and how it can be rectified?

Which PID is using the most cpu?
top will show you.

Examining the php error log I find an entry just before the lock up that says:

Php Warning: file_put_contents(): Only 0 bytes out of 1013 bytes written, possibly out of free disk space (entry was terminated before completing the php file that was running)

Followed by:
PHP Fatal Error: Out of memory (allocated 14155776) (tried to allocate 1572935 bytes)

The USB stick is an 8GB stick all dedicated to antiX, so there should be ample disk space available.

In looking at the display from top it did not appear that any process was consuming much cpu time. X was the first listing in top, which I think displays the processes in the order of their cpu usage.

frank wrote:Examining the php error log I find an entry just before the lock up that says:

Php Warning: file_put_contents(): Only 0 bytes out of 1013 bytes written, possibly out of free disk space (entry was terminated before completing the php file that was running)

Followed by:
PHP Fatal Error: Out of memory (allocated 14155776) (tried to allocate 1572935 bytes)

The USB stick is an 8GB stick all dedicated to antiX, so there should be ample disk space available.

It looks like this is more of a memory problem than a disk problem. I don't understand why it says 14 Meg were allocated when it was trying to allocate 1.5 Meg. Still, 1.5 Meg is a large chunk of RAM and it is possible there were no more free chunks of memory that large. Do a Google(/proc/buddyinfo) for more information. That file can tell you if this is the problem. Also take a look in the output of the"dmesg" command for more details.

It is possible that running with dynamic root persistence contributes to the problem. You can rule this out by running with static root persistence (ideally with a usb-3.0 LiveUSB or with a frugal install on a fast drive). This will run slower but it won't touch the RAM.

Writing a lot of data to the root file system when using dynamic root persistence can eat up much more RAM than is shown in many monitoring programs. The problem is that the RAM used by dynamic root persistence to hold file system changes uses tmpfs which counts as"cached" memory and most monitor programs assume all of the cached memory can be freed if needed. But cached memory used for tmpfs cannot be freed (or your tmpfs would lose files). To find out the true story, try these two commands:The first line frees up as much cached memory as possible. What is left in the"cached" column cannot be freed. The true amount of free memory is listed in the first line in Megabytes. Ignore the"-/+ buffers/cache" line. If you are running out of RAM and a full install or static root persistence aren't good options for you then you *might* be able to get around the problem by adding more swap space. You have to be careful though because if you are constantly swapping then that is going to slow your system to a crawl.

If you have installed a lot of stuff and haven't done a live-remaster then doing a live-remaster and a reboot will free up RAM. A live-remaster will help you with things you've added to the root partition that don't need to change (much). If you need a lot of storage space on the root partition that changes as you run then try to move it somewhere else. For example, you can make a directory on the LiveUSB (mounted at /live/boot-dev) and then make a symlink from your storage directory to the directory you made. Or you can enable home persistence and make a symlink from your storage directory to a directory under /home. Storing file system changes in RAM (which is what happens with no root persistence and with dynamic root persistence) can be very fast but the trade off is it uses up RAM.

Thank you for your assistance. Yes, it appears that the system is running out of ram. I have installed several packages, so I took your advice about creating a remastered version of the usb. Before doing so, I rebooted the computer to give it a fresh version and when I attempted the remastering it failed. I tried both compression types offered and both failed to write the .tmp file. I'm not sure how to move forward at this point???

run with the"static" root persistence options. its slower, but it doens't load the root persistence file into ram, so you shouldn't run out.

if the remaster failed, its likely due to the size of the usb stick. How big is the stick, and how big did you make the persistence files.

I'm using an 8GB usb stick. When I enter the remaster app it says the estimated new size is 716 meg (xz) or 873 meg (gzip). It shows the boot device having 7381 meg total space with 5566 meg free and the est. remaining using gzip of 4693. When I go into the app it asks for a password but does not accept the root password, only the password for the user account. Is that normal? Logging out of the user account and trying to log in to the system as root leads nowhere. I suspect this may be what is causing the difficulty with the remastering, but I'm not sure. I'm relatively inexperienced as a linux administrator, so it may be my lack of understanding about why the password for the root account is not accepted when entering the remastering app.

frank wrote:I'm using an 8GB usb stick. When I enter the remaster app it says the estimated new size is 716 meg (xz) or 873 meg (gzip). It shows the boot device having 7381 meg total space with 5566 meg free and the est. remaining using gzip of 4693. When I go into the app it asks for a password but does not accept the root password, only the password for the user account. Is that normal? Logging out of the user account and trying to log in to the system as root leads nowhere. I suspect this may be what is causing the difficulty with the remastering, but I'm not sure. I'm relatively inexperienced as a linux administrator, so it may be my lack of understanding about why the password for the root account is not accepted when entering the remastering app.

antix is using"sudo" mode for gksu, so the user account should be the correct one.

run the remaster from"static" persistence mode.

Thanks for the help. It worked ... using static root persistence allowed the remastering to complete. I chose personal remaster to get the system that I had configured will all settings intact and when I rebooted and tried to use the system apache2 would not start. The diagnositc said it could not find /var/log/apache2/. I also checked the mysql service and it was stopped so I tried starting it and it failed, so apparently the remastering did not capture the full system correctly. Or maybe I did something wrong??? I'm not sure what else is not working as my purpose for this system is centered around apache2 and mysql. I looked for an option to rollback to the former configuration, but could not find it. Where is that located?

use"rollback" as a boot cheatcode.

However, consider that its likely the system just didn't save that particular directory (log files are usually excluded to save space). you could just remake it the directory.

Thanks again. I did rollback to the original version. Also running with static persistence has resolved the origiinal problem I was having. Now I'd like to remaster the usb stick, but I don't know all the add-backs that I'll need to manually make to get all the installed software working again. It seems like the apache logs is just one of the places that were omitted since mysql also failed to start. Is there anyway that all of the required directories for the installed software can be retained during the remastering process. Obviously, the log files themselves need not be retained, but the directories should be so that when attempting to write logs, the services don't fail. At least I'd like to make a backup of this working version now so that if something goes wrong in the future I can roll back to the backup and move forward from there. Any ideas on how to do that? Thanks again, you all have been very helpful, and this is a very nice distribution!

I found the luckybackup tool but it says it lets you modify essential parts of the system. Obviously that is not what I'm looking to do, so I was reluctant to use it at this point. I will do more research on that tool now.

frank wrote: Is there anyway that all of the required directories for the installed software can be retained during the remastering process. Obviously, the log files themselves need not be retained, but the directories should be so that when attempting to write logs, the services don't fail. At least I'd like to make a backup of this working version now so that if something goes wrong in the future I can roll back to the backup and move forward from there.

Unfortunately, we don't know every required file and directory for every Debian package.

On the plus side, you (as the user with root privileges) can control which files get excluded when we do a live-remaster and a persist-save by editing text files in /usr/local/share/excludes. For example in the file: /usr/local/share/excludes/live-remaster-exclude.list is this line:This line excludes everything in the /var/log/ directory EXCEPT the samba/ directory. You could try either deleting this line or commenting it. This will save all log files and that might be enough to let mysql work after a remaster.

Obviously, the log files themselves need not be retained, but the directories should be so that when attempting to write logs, the services don't fail.

Thank you for this excellent idea! We might be able to generate the /var/log/ portion of the exclude list with a find command so we exclude files but not directories. In the meantime, I hope that you will be able to get by with editing the exclude list manually.